Method, apparatus and system for image processing

ABSTRACT

An image processing device, including an image receiver operable to receive a first image; an image converter operable to convert the first image to a second image, the second image being a smaller-sized representation of the first image; an image pre-processor operable to perform an imaging effect on the second image to form a third image, the imaging effect being defined by an imaging effect parameter; a first output device operable to output the third image for display; and a second output device operable to output the first image and the imaging effect parameter.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is based on PCT filing PCT/GB2014/051422 filedMay 9, 2014, and claims priority to United Kingdom applicationGB1312028.2 & GB1312029.0 filed on 4 Jul. 2014, the contents of each ofwhich are incorporated herein by reference in their entirety.

BACKGROUND

Field of the Disclosure

The present invention relates to a method, apparatus and system forimage processing.

Description of the Related Art

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thebackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, are neitherexpressly or impliedly admitted as prior art against the presentinvention. Currently, it is possible to add visual effects to imagesusing computer software. For example, it is possible to emulate thecapability of a shift lens by applying a geometric transformation to animage using Adobe® Photoshop®. This is known as shift correction.However, there are two problems with this. Firstly, a user must captureand store an image before applying the correction. This can cause severeproblems during capturing images because the application shiftcorrection may crop the image. As the user does not know in advance whatparts of the image will be cropped, application of the shift correctionmay crop a relevant part of the image.

Secondly, the application of visual effects such as shift correction aregenerally only performed with still images. This is due to a lack ofsuitable methods for performing such effects on moving images.Furthermore, if such effects are applied to moving images, such asduring film-making, the problem noted above regarding inadvertentcropping of images is a severe problem. It is very expensive, andsometimes impossible, to re-shoot a scene where inadvertent cropping ofimages in a scene has taken place. It is an aim of embodiments of thedisclosure to address at least these issues.

SUMMARY

In one aspect, the present invention provides an image processingdevice, comprising image receiver circuitry configured to receive afirst image; image converter circuitry configured to convert the firstimage to a second image, the second image being a smaller-sizedrepresentation of the first image; image pre-processor circuitryconfigured to perform an imaging effect on the second image to form athird image, the imaging effect being defined by an imaging effectparameter; first output device circuitry configured to output the thirdimage for display; and second output device circuitry operable to outputthe first image and the imaging effect parameter.

The foregoing paragraphs have been provided by way of generalintroduction, and are not intended to limit the scope of the followingclaims. The described embodiments, together with further advantages,will be best understood by reference to the following detaileddescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 describes a user capturing an image of a tall object using acamera;

FIG. 2 describes an image sensor of the camera being tilted with respectto the surface of the tall object;

FIG. 3 describes an image captured by the camera in which vertical linesof the tall object appear to incorrectly converge;

FIG. 4 describes a shift correction imaging effect which can be appliedto the image in order to correct for the convergence of the verticallines;

FIG. 5 describes an imaging processing device for generating a previewimage with shift correction applied according to an embodiment of thepresent invention;

FIG. 6 describes a camera comprising the image processing device forgenerating the preview of the shift corrected image;

FIG. 7 describes an image processing device for performing the shiftcorrection applied to the preview image on the original image accordingto an embodiment;

FIGS. 8A-8D describe various methods of connecting image processingdevices to form a system according to embodiments of the presentinvention;

FIG. 9 describes a table recording a list of imaging effect parameters;

FIG. 10A describes a process for generating a preview image with shiftcorrection applied;

FIG. 10B describes a process for performing the shift correction on theoriginal image;

FIG. 11 describes a user capturing images of a tall object in a livecamera shoot;

FIG. 12 describes capturing different images of the tall object with thecamera in different positions;

FIG. 13 describes the different images of the tall object that have beencaptured;

FIGS. 14A-14C describe the angular offset of lines in a captured imagewith respect to the vertical and horizontal;

FIGS. 15A-15B describes a process for automatically detecting a subsetof straight lines in a captured image and shift correcting the imagebased on the subset, according to an embodiment of the presentinvention;

FIG. 15C describes the calculation of the angles of a detected line inan image with respect to the vertical and horizontal;

FIG. 16 describes manually selecting or deselecting a significant linefor calculating shift corrections for the image;

FIG. 17 describes an image which has been shift corrected according toeither automatic or manual detection of a significant line;

FIG. 18 describes a process for processing a captured image according toan embodiment;

FIG. 19 describes the marking of certain images in a sequence of imageswhich are to be shift corrected;

FIG. 20 describes the shift correction of the unmarked images byinterpolating the shift correction of the marked images;

FIG. 21 describes correcting the interpolated shift correction of themarked images to take into account changes in the zoom level of thecamera;

FIG. 22 describes a tablet computer which may be used for processingcaptured images according to an embodiment; and

FIG. 23 describes a camera and tripod arrangement which may be used forcapturing images according to embodiments.

DESCRIPTION OF THE EMBODIMENTS

In one aspect, the present invention provides an image processingdevice, comprising an image receiver configured to receive a firstimage; an image converter operable to convert the first image to asecond image, the second image being a smaller-sized representation ofthe first image; an image pre-processor operable to perform an imagingeffect on the second image to form a third image, the imaging effectbeing defined by an imaging effect parameter; a first output deviceoperable to output the third image for display; and a second outputdevice operable to output the first image and the imaging effectparameter.

Advantageously, since the second image is a smaller-sized representationof the first image, an imaging effect can be applied to the second imagewith less processing when compared to applying the imaging effect to thefull-size first image, which can be a processor intensive task. A useris able to thus quickly and easily preview the imaging effect with thepresent invention. At the same time, because the full-size first imageis output with the imaging effect paramater, which defines the imagingeffect, that same imaging effect can then be applied to the full-sizefirst image at a later time. A high quality, full resolution image withthe imaging effect applied can thus still be obtained.

In embodiments, the second image may be one of a compressed version ofthe first image or a lower resolution version of the first image.Advantageously, both of these image types can have an imaging effectapplied to them with less processing when compared to the full-sizefirst image.

In embodiments, the second output device may output the first image andimaging effect parameter to a storage medium. Advantageously, thisallows the first image and imaging effect parameter to be retrieved andprocessed at a time convenient to the user.

In embodiments, the second output device may transmit the first imageand the imaging effect parameter to a remote device. Advantageously,this allows the first image and imaging effect parameter to be passed toa device which may be more suitable for the processor intensive task ofperforming the imaging effect on the full-size first image. The workflow of performing the imaging effect on the full-size first image isthus made more efficient.

In embodiments, the first image and imaging effect parameter may betransmitted to the remote device over a computer network.Advantageously, this allows the first image and imaging effect parameterto be transmitted using existing computer network infrastructure,improving the convenience for the user.

In embodiments, the imaging effect may be a shift correction effect andthe imaging effect parameter may be a shift correction parameter. Theremay be a shift correction controller operable to generate the shiftcorrection parameter. Advantageously, this allows the user to preview ashift correction effect as an imaging effect on the second image. Thisaids the user in image composition, as it allows the user to be aware ofany sections of an image which may become cropped following a shiftcorrection effect being applied.

In embodiments, the shift correction parameter may be an angle by whichthe second image must be digitally rotated in three-dimensional space inorder to achieve the shift correction effect. Advantageously, this givesa direct numerical value which indicates to an image processor theextent to which an image must be rotated in order to perform the shiftcorrection. No additional processing thus needs to be performed on theshift correction parameter before the shift correction effect can beapplied, thus reducing the total amount of processing required.

In embodiments, the shift correction controller may receive an inputfrom a user to select a value of the angle of the shift correctionparameter to be generated. Advantageously, this gives direct control tothe user as to the extent of shift correction to be applied to an image.

In embodiments, the first image may be captured by a camera and thevalue of the angle of the shift correction parameter may be selected bythe shift correction controller using data from one of an accelerometeror gravimeter associated with the camera. Advantageously, this allowsthe extent of shift correction to be applied to an image to be decidedautomatically without the need for input from the user.

In embodiments, the image receiver may receive a plurality of firstimages at a predetermined frame rate, and a time between the imagereceiver receiving each first image and the first output deviceoutputting a respective third image for display may be less than orequal to the time period between successive first images.Advantageously, this allows an imaging effect on each image in asequence of captured images to be previewed in real time. This aids theuser in composing a sequence of images displayed in a digital viewfinderor captured as a video.

In another aspect, the present invention provides an image processingdevice, comprising a receiver operable to receive an image and animaging effect parameter, the imaging effect parameter defining animaging effect to be performed on the image; an image processor operableto perform the imaging effect on the image, using the imaging effectparameter, to produce a final image; and a final image output deviceoperable to output the final image.

Advantageously, an imaging effect may thus be applied to an image usingnothing except the predetermined imaging effect parameter associatedwith the image. No input from the user is required, improving theconvenience for the user.

In embodiments, the receiver may receive the image and imaging effectparameter from a remote device. Advantageously, this allows the firstimage and imaging effect parameter to be received from a device whichmay be less suitable for the processor intensive task of performing animaging effect on the image. The work flow of performing the imagingeffect on the image is thus made more efficient.

In embodiments, the image and imaging effect parameter may be receivedover a computer network. Advantageously, this allows the image andimaging effect parameter to be transmitted using existing computernetwork infrastructure, improving the convenience for the user.

In another aspect, the present invention provides a system comprising afirst image processing device, comprising an image receiver operable toreceive a first image; an image converter operable to convert the firstimage to a second image, the second image being a smaller-sizedrepresentation of the first image; an image pre-processor operable toperform an imaging effect on the second image to form a third image, theimaging effect being defined by an imaging effect parameter; a firstoutput device operable to output the third image for display; and asecond output device operable to output the first image and the imagingeffect parameter; and a second image processing device, comprising areceiver operable to receive the first image and the imaging effectparameter; an image processor operable to perform the imaging effect onthe first image, using the imaging effect parameter, to produce a finalimage; and a final image output device operable to output the finalimage.

Advantageously, the first image processing device allows a user toquickly and easily preview the imaging effect on the second image, whichis a smaller-sized representation of the first image. At the same time,the full-size first image and imaging effect parameter, which definesthe imaging effect, are passed to the second image processing device, sothat the imaging effect can be applied to the full-size first image. Ahigh quality, full resolution image with the imaging effect applied canthus still be obtained.

In embodiments, the first and second image processing devices may beremotely located from each other, and the second output device of thefirst image processing device may transmit the first image and imagingeffect parameter to the receiver of the second imaging processingdevice.

Advantageously, this allows the first and second image processingdevices to be or to be comprised within suitable devices within suitablelocations. For example, the first image processing device could becomprised within a camera located on a film set and the second imageprocessing device could be comprised within a studio control room in adifferent location. The different tasks of the first and second imageprocessing devices are thus delegated to the most suitable devices andlocations, improving the efficiency of the image processing work flow.

In embodiments, the first image and imaging effect parameter may betransmitted over a computer network. Advantageously, this allows thefirst image and imaging effect parameter to be transmitted usingexisting computer network infrastructure, improving the convenience forthe user.

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views.

FIGS. 1-4 demonstrate an example of a visual effect, also known as animaging effect, which may be applied to images in embodiments of thepresent invention. Specifically, FIGS. 1-4 demonstrate an imaging effectwhich corrects for the convergence of vertical lines in the image.

FIG. 1 illustrates a user 100 using a camera 105 to capture an image ofan object 110 such as a building. The object 110 comprises a surface 115which extends in a substantially vertical direction from the ground 120.It is noted that in order to capture the entirety of the object 110 inan image, the user 100 is required to tilt the camera 105 upwards.

FIG. 2 illustrates the tilting of the camera 105 in more detail. It canbe seen that the camera 105 comprises an image sensor 200, such as acharge-coupled device (CCD) or complementary metal-oxide-semiconductor(CMOS) sensor, which in combination with lens 210, allows a digitalimage of the object 110 to be captured. Specifically, the lens 210focuses light received from the object 110 onto an image plane 205, andthe image sensor 200 is positioned so as to be aligned with the imageplane 205, thus allowing an image of the object 110 formed at the imageplane 205 to be captured. It can be seen that because the camera 105 istilted, the image plane 205 is not parallel to the plane defined by thesurface 115 of the object 110.

The effect of the image plane 205 not being parallel to the planedefined by the surface 115 is shown in FIG. 3. FIG. 3 shows a capturedimage 300 of the object 110 as viewed from the perspective of the camera105 when the camera is in the position illustrated in FIGS. 1 and 2. Itcan be seen that the surface 115 is bounded by two straight lines 305,310 which form edges of the object 110. In reality, the lines 305, 310extend in a vertical direction from the ground 120 and are parallel toeach other. However, because of the tilt of the camera 105, the lines305, 310 in the image appear non-parallel. Rather, the lines 305, 310appear to converge. This can be an undesirable effect, because itincorrectly makes the object 110 in the image 300 appear to be leaningbackwards.

Such an effect can be avoided by using a special shift lens with thecamera 105, which allows the image plane to be shifted with respect tothe image sensor 200 whilst the camera 105 remains stationary. Thisallows the entirety of the object 110 to be captured by the camera 105without having to tilt the camera 105 upwards. The image plane 205 canthus be kept parallel to the plane defined by the surface 115 of theobject 110, and therefore the convergence of the lines 305, 310 in theimage 300 is avoided. Special shift lenses are, however, very expensiveand may have other associated problems. Fortunately, the convergence ofthe lines 305, 310 can be corrected digitally instead duringpost-processing, by performing an imaging effect on the image 300.

The result of performing an imaging effect to correct for theconvergence of lines 305, 310 on the image 300 is illustrated in FIG. 4.Here, a corrected version of the image 300′ is shown. It can be seenthat the image has been corrected so that lines 305, 310 are nowparallel and do not converge. Specifically, the image 300′ has beenformed by digitally rotating the image 300 in three-dimensional (3D)space so as to change the perspective of the image. In FIG. 4, the axis400 about which the image 300 was rotated to form image 300′, togetherwith the direction of rotation 405, is shown. It is noted that theposition of the axis 400 could be vertically offset in an upwards ordownwards direction and still define an axis about which the image 300can be rotated for perspective correction.

Digitally rotating an image in 3D space to change the perceivedperspective of the image is known in the art. For example, it is aneffect that can be applied to an image using imaging editing programssuch as Adobe® Photoshop®.

For any given image 300, the image may be corrected by considering oneof the lines 305, 310 and changing the perspective of the image 300until that line becomes truly vertical. Example methods of analysing thelines in a captured image for the purpose of line convergence correctionare discussed later. Because of the rotation of the image 300 to formthe corrected image 300′ and the fact that it is desirable for thecorrected image 300′ to occupy the same rectangular shape with the samedimensions as that of the original image 300, it can be seen that thecorrected image 300′ comprises blank portions 410, 415 and croppedportions 420, 425. The blank portions 410, 415 arise in regions of theimage which appear to move away from view as the image is rotated. Onthe other hand, the cropped portions 420, 425 arise in regions of theimage which appear to move closer as the image is rotated and whichexceed the boundaries of the rectangular frame of the original image. Asdiscussed above, these portions, in particular the cropped portions 420,425 can cause problems to the user 100 capturing the original image 300,since the user 100 may be unsure as to which areas of the image 300 maybe lost once the image is corrected during post-processing.

The problem in which the user is unsure as to which areas of the image300 may be lost may be made even worse if the corrected image 300′ isfurther cropped and scaled so as to remove the blank regions 410, 415and restore the image so that it once again becomes rectangular inshape. For example, as illustrated in FIG. 4, the corrected image 300′could be cropped along the dotted lines 430, 435 so that portions 440,450 containing the blank regions 410, 415 are removed and an image witha rectangular shape is obtained. It is very difficult for a user topredict the extent of the regions 440, 450 which will be cropped fromthe image during post-processing, and there is therefore a risk thatimportant features in the regions 440, 450 of the image may be lost.

If the user 100 is capturing a still image 300, it is thereforedesirable to provide the user 100 with a preview of the image, with theconvergence correction applied, before the original image 300 iscaptured and stored. That is, the user is provided with a preview of thecorrected image 300′ in real time, before the original image 300 iscaptured and stored. The preview image may be displayed to the user 100in a live-view format using, for example, a screen on the camera 105which acts as a digital viewfinder. Similarly, if the user 100 iscapturing a video, in which each frame of the captured video isequivalent to an image 300, it is desirable to provide the user with apreview of the video, with the convergence correction applied to eachframe, whilst the video is being captured and stored. That is, the useris provided with a preview of the corrected image 300′ in real time foreach captured image 300 which forms a frame of the video. Again, thepreview video may be displayed to the user 100 in a live-view formatusing, for example, a screen on the camera 105 which acts as a digitalviewfinder.

The problem with performing the correction on a captured image 300 inreal time is that this is a very processor intensive task. This isparticularly true for the high resolution photo and video cameras whichexist today. There is therefore a danger that the processing forcorrecting a particular image 300 will take too long, resulting in atime-lag between the user capturing an image 300 and the user being ableto view a preview of that image with the correction applied. This canresult in the user composing an image incorrectly during image captureand not being able to realise that the composition is incorrect untillater on.

Although the image may be re-captured in certain circumstances, this isoften not possible. For example, when filming a fast-moving sport suchas soccer, a user may have only one chance to capture an image of aparticular event such as a goal being scored. The present invention aimsto alleviate this problem, so that captured high resolution images canhave correction applied to them and can be previewed by the user inreal-time, without a significant time lag.

FIG. 5 illustrates an image processing device 500 according to anembodiment of the present invention. The device 500 comprises a receiver505 which receives an image 300 from an image source. In this example,the image 300 has been captured using the camera 105. However, it willbe appreciated that the image 300 may be obtained from another source.For example, the image 300 may have been captured at an earlier time andmay be stored on a storage device such as a hard disk drive or a flashmemory.

The image 300 is then passed to both a converter 510 and to animage/parameter output 515. Each of these paths of the image 300 areconsidered separately. In FIG. 5, solid lines indicate the path of theimage 300 which is being processed. Dashed lines indicate the path ofnon-image, parameter data related to the image 300. The dot-dash lineindicates an optional path for the image 300.

When the image 300 is received at the converter 510, processing isperformed on the image 300 so that it is converted to a smaller-sizedrepresentation of itself. For example, if the image 300 is in a standarduncompressed data format, then the image 300 could be easily convertedto a lower resolution image, so that it is represented by a smallernumber of pixels. This can be achieved by interpolating pixels of theimage 300 at its original resolution. The smaller-sized representationof the image 300 is smaller-sized in that the same image is representedby a smaller amount of information. For example, a 30 MB image can beconverted to a 4 MB image. Because the resulting smaller-sizedrepresentation of the image 300 (which may be referred to as thesmaller-sized image) is represented by a smaller amount of information,less processing is required for applying an imaging effect, such as lineconvergence correction, to the smaller-sized image when compared to thatrequired for the original image 300.

Once the converter 510 has converted the image 300 to a smaller-sizedimage, the smaller-sized image is passed on to the image pre-processor520. Here, a suitable imaging effect is applied to the smaller-sizedimage. So, for example, if line convergence correction is required, suchas that illustrated in FIG. 4, the image pre-processor 520 performs theline convergence correction processing on the smaller-sized image. Lineconvergence correction may also be referred to as shift correction, dueto it traditionally being performed using a shift lens.

In the embodiment shown in FIG. 5, shift correction processing isperformed on the smaller-sized image. In order to perform the shiftcorrection, the image pre-processor requires extra information in orderto determine how to correct the image. This extra information isprovided in parameter form by shift correction controller 530. The shiftcorrection controller 530 provides one or more imaging effectparameters, depending on what information is required by the shiftcorrection method implemented by the image pre-processor 530. So, forexample, if the shift correction method illustrated in FIG. 4 is used,in which the image is rotated in 3D space to compensate for the tilt ofthe image sensor 200, then the shift correction controller 530 mayprovide a numerical value indicating the angle by which the image is tobe rotated as the imaging effect parameter.

The shift correction controller 530 may compute the imaging effectparameter in any suitable way. For example, the shift correctioncontroller 530 could comprise a manual controller (not shown) throughwhich the user is able to manually select the extent of shift correctionof the image. So, for example, if the shift correction method of FIG. 4is used, then the user may use a manual controller of the shiftcorrection controller 530 to select an angle of rotation of the image.The manual controller could comprise a dial, button or touch screen, forexample.

Alternatively, the shift correction controller 530 could automaticallygenerate an imaging effect parameter for the shift correction. Animaging effect parameter may be automatically generated using anysuitable method, such as through the use of an accelerometer orgravimeter in the camera 105 (not shown) which captures the originalimage 300 or through the use of edge detection (also known as linedetection) in the image. In the former example, a gravimeter maygenerate an angle indicating the amount of tilt of the image sensor 200with respect to the gravitational vertical. This angle may then be usedto determine the extent of shift correction on the image, with largerangles indicating that a larger amount of shift correction is required.In the latter example, a vertical or horizontal edges which are detectedin the image may be used as a reference for performing the shiftcorrection. An imaging effect parameter, such as an angle indicating therotational offsets of the detected edges with respect to the truevertical or horizontal, may then be generated by the shift correctioncontroller 530. This allows shift correction to be performed on theimage by rotating the image in 3D space until the rotational offset ofthe detected edge is removed. The edge detection correction technique isdescribed in more detail later on.

In the embodiment of FIG. 5, the edge detection processing is performedon the smaller-sized image generated by the converter 510, which reducesthe processing effort required when compared to performing the edgedetection processing on the larger original image 300. It isconceivable, however, that the edge detection could instead be performedon the larger original image 300.

After the smaller-sized image has been processed, the smaller-sizedimage is passed to display output 525. The display output is operable tooutput the smaller-sized image, with the imaging effect applied to adisplay device (not shown). The display device may be any suitabledevice for displaying the smaller-sized image as a preview image to theuser. For example, the display device could be a computer monitor, atelevision or a preview screen integrated as part of the camera 105 ormobile computing device. When an image 300 is received by the receiver505, it is also sent to the image/parameter output 515. The image 300sent to the image/parameter output 515 is the full-sized, original imagereceived by the receiver 505. The one or more imaging effect parametersgenerated by the shift correction controller 530 for the smaller-sizedrepresentation of the image 300 are also sent to the image/parameteroutput 515. The image/parameter output 515 then outputs the image 300and the associated imaging effect parameter(s). Although the displayoutput 525 and image/parameter output 515 are illustrated as separatecomponents in the embodiment of FIG. 5, they may also be comprisedwithin a single output. In this case, the preview image data and thefull image/parameter data may be multiplexed together. In someembodiments imaging effect parameters might be applied to the imagebefore outputting the combined image.

As will be explained later, the image 300 and its associated imagingeffect parameter(s) may be stored together in a storage medium forfurther processing at a later time. Alternatively, they may be senttogether to a remote device for further processing. The imaging effectparameter(s) associated with the image 300 defines the shift correctionthat is to be applied to the image 300. This will be the same shiftcorrection that was applied to the smaller-sized representation of theimage 300 by the image pre-processor 520. The further processing of theimage 300 and its associated imaging effect parameter(s) involvesperforming shift correction processing on the image 300 so that thefull-sized, original image 300 may be shift corrected to match thecorrection of the smaller-sized preview image.

In embodiments, the receiver 505 may receive a sequence of images at apredetermined frame rate. For example, the images received at thereceiver 505 may be captured from the camera 105 at a predeterminedframe rate and provided to the receiver 505 in real time. The sequenceof images is then pipelined through the device 500. The frame rate maybe a frame rate that is used for video capture, for example, 24, 50, 60,100 or 200 frames per second.

The sequence of captured images may be stored as a video. In this case,for each image 300 in the sequence of images, the image 300 is sent tothe converter 510 to be converted to a smaller-sized image. Shiftcorrection is then performed on the smaller-sized image by the imagepre-processor 520, using one or more imaging effect parameters generatedby the shift correction controller 530. The smaller-sized image, withshift correction applied, is then output by the display output 525. Atthe same time, the original image 300 is sent to the image/parameteroutput 515. The one or more imaging effect parameters generated by theshift correction controller 530 are also sent to the image/parameteroutput 530. The original image 300 is then output, together with the oneor more imaging effect parameters, to a storage medium or to a remotedevice for shift correction to occur at a later time.

Alternatively, the sequence of captured images may simply be displayedto the user without necessarily being stored. In this case, a displaywhich receives images from the display output 525 may act as a live-viewelectronic viewfinder of the camera 105. For each image 300 in thesequence of images, the image 300 is again processed by the converter510 and image pre-processor 520 and output by the display output 525.Also, the original image 300 and the one or more imaging effectparameters are again sent to the image/parameter output 530. This time,however, the original image 300 and associated imaging effectparameter(s) are not automatically output by the image/parameter output515. Rather, the image 300 and imaging effect parameter(s) aretemporarily held in a memory of the image/parameter output 515 (notshown).

The image 300 and imaging effect parameter(s) may be output by theimage/parameter output 515 in response to a command issued by the user.This command could be issued by the user by pressing a shutter button onthe camera 105, for example. If the command is issued by the user, thenthe image that is currently being output by the display output 525 (inpreview form) and that is currently also being held in the memory of theimage/parameter output 515 (in full-size form) is output by theimage/parameter output 515 to a storage device and/or a remote device,together with the associated imaging effect parameter(s). Alternatively,if the command is not issued by the user before the next image in thesequence is received by the image/parameter output 515, then the image300 and imaging effect parameter(s) are overwritten in the memory of theimage/parameter output 515 by the data associated with the next image.

Once the command has been issued by the user and the current image hasbeen output by the image/parameter output 515, the device can return toits original state in which the preview images are output for displaybut in which the original images 300 and parameter(s) are not stored.This allows a still image to be captured. Alternatively, the device canchange, in response to the command being issued by the user, to a videostorage mode. Here, for all subsequently received images, both a previewimage is displayed and the original image 300 and parameter(s) arestored. The device thus allows a video to be recorded from the momentthat the command is issued by the user.

For both the video capture and live-view embodiments, for each image 300in the sequence, the amount of information used to represent thesmaller-sized image should be chosen such that the image pre-processor520 is able to apply the imaging effect in real time at the frame rateat which the images are received. That is, the image pre-processor 520should be able to apply the imaging effect to each image in the sequencein a sufficiently small amount of time such that the time-lag betweenthe receiver 505 receiving an image 300 and the display output 525outputting a preview image is less than or equal to the time periodbetween successive image frames. This allows a user to preview eachshift corrected image in the captured image sequence in real time,allowing them to correctly compose still images or video images whichhave been shift corrected and which may therefore have blank or croppedimage portions. The display of the preview image is illustrated in FIG.6, which shows the camera 105 when it comprises the image processingdevice 500 (not shown in FIG. 6). Here, the camera 105 also comprises anintegrated display 605 which receives the processed preview images fromthe display output 525. A preview of the shift corrected image 300′ ofthe object 110, with blank portions 410, 415 and cropped portions (notshown), is shown on the display 605. This is a live-view preview imageof the object 110, in which images are being captured by the camera 105and sent to the image processing device 500 for processing and displayin real time. Because a preview of the corrected image 300′ can be seen,the user is able to compose the image correctly by ensuring that allobjects they wish to capture do not fall within the cropped portions.

The camera 105 comprises shutter button 610. As described above, theshutter button 610 is pressed to issue a command to store the full-sizecaptured image 300 and associated imaging effect parameter(s) related tothe preview image currently being displayed on the display 605. Afterthe shutter button 610 has been pressed, the camera 105 can continue todisplay preview images only, that is, without storing the images andparameter data. Alternatively, the camera 105 can enter a video mode andcan both display preview images and store the images and parameter data.

It will be appreciated that although FIG. 6 shows a consumer camera 105in which the image processing device 500 is integrated, the imageprocessing device 500 could also be used as a standalone unit. Forexample, the image processing unit 500 could be a standalone unitdesigned for use with a professional video camera such as a Sony PMW-F55or PMW-F5 CineAlta 4K camera. In this case, video images captured by thecamera are fed to the receiver 505 of the device 500. The two separateresulting video feeds, one being output by the display output 525 andthe other being output by the image/parameter output 515, may then berespectively fed to a preview screen of the camera and to a videostorage device of the camera. The preview screen and video storagedevice each may or may not be comprised within the camera itself.

Alternatively the image processing device 500 may be integrated within atablet (such as the Sony Xperia® Z tablet) or smartphone (such as theSony Xperia® Z smartphone). In this case, the image processing device500 may be implemented in the tablet or smartphone as software, forexample as a downloadable software application (or “App”).

Once the full-size image 300 and associated imaging effect parameter(s)have been stored or sent to a remote device, the imaging effect whichwas performed on the smaller-sized image can be applied to the full-sizeimage 300. This can be achieved using image processing device 700, asillustrated in FIG. 7. Again, in FIG. 7, solid lines indicate the pathof the image 300 which is being processed. Dashed lines indicate thepath of non-image, parameter data related to the image 300.

The full-size original image 300 and associated imaging effectparameter(s) are received at the input 705. The input 705 may also bereferred to as a receiver. The image 300 and parameter(s) may bereceived from a storage device (not shown) or they may be receiveddirectly from the image/parameter output 515 of the image processingdevice 500. In the latter case, the image processing device 700 may actas a remote device to which the image 300 and parameter(s) are sent. Theimage 300 and parameter(s) are then sent to image processor 710. Theimage processor 710 performs an imaging effect on the image 300. This isthe same imaging effect which was performed on the smaller-sizedrepresentation of the image 300 by the image pre-processor 520. So, forexample, the imaging effect may be a shift correction of the image 300.The imaging effect is performed on the image according to the imagingeffect parameter(s). Once the imaging effect has been applied to theimage 300, the processed image 300′ is passed to the output 715. Theoutput 715 may then output the processed image 300′ to a device such asa storage device or to a display.

Through the combination of the image processing devices 500, 700, a usermay therefore capture an image 300 and preview an imaging effect on asmaller-sized representation of that image in real time. This allows theuser to compose the image 300 correctly as it is captured. The sameeffect can then be applied to the full-size version of the image 300 ata later time. Full-size, correctly composed images 300′ with an imagingeffect applied can therefore be obtained more easily by the user withembodiments of the present disclosure.

The image processing devices 500, 700 can be combined using any suitablemethod which allows image and parameter data output by theimage/parameter output 515 of the device 500 to be received by the input705 of the device 700. Example embodiments are illustrated in FIGS.8A-8D.

In FIG. 8A, the image processing device 500 is comprised within a camera105. The full-size images 300 and parameters are transmitted to theimage processing device 700 via a direct connection 810. The directconnection 810 may be a wired or wireless connection. Examples ofsuitable wired connections could include a USB, FireWire, HDMI orethernet connection. Examples of suitable wireless connections couldinclude a Wi-Fi, Bluetooth, Infra-Red (IR), Radio Frequency (RF) oroptical connection. A wireless connection could be established using theDigital Living Network Alliance® (DNLA) wireless standard. In FIG. 8B,the image processing device 500 is comprised within a camera 105. Thecamera 105 also comprises a portable storage medium drive 815 into whicha portable storage medium 825 may be inserted. The full-size images 300and parameters are recorded to the portable storage medium 825. Theportable storage medium 825 can then be removed from the portablestorage medium drive 815 of the camera 105 and inserted into a portablestorage medium drive 820 of the image processing device 700, thusallowing the device 700 to have access to the full-size images 300 andparameters. Any suitable portable storage medium 825 could be used.Examples include an optical disc, such as a compact disc (CD), digitalversatile disc (DVD) or Blu-Ray disc (BD), or a portable flash memorydevice, such as a Memory Stick, Secure Digital (SD) card or CompactFlash(CF) card.

In FIG. 8C, the image processing device 500 is comprised within a camera105 which is operable to connect to a computer network 830. The imageprocessing device 500 is able transmit the full-size images andparameters to the image processing device 700 over the network 830. Thenetwork 830 could be a local computer network or it could be theinternet.

In the embodiment of FIG. 8C, there could be a plurality of imageprocessing devices 700 which together process a stream of full-sizeimages 300 received from the image processing device 500. In this case,a plurality of images received from the device 500 could be processed inparallel, with each of the images being processed by a different device700. Advantageously, a user is thus able to capture images using thecamera 105 and view a preview of the captured images, with an imagingeffect applied, in real time. The workload of then applying the imagingeffect to each of the captured full-size images is then offloaded to theplurality of devices 700 via the network 830. Such an arrangement may beknown as cloud processing of the captured images.

It is noted that although the image processing device 500 is comprisedwithin a camera 105 in FIGS. 8A-8C, the device 500 could instead be astandalone device. In this case, it would be the device 500 itself whichhas the direct connection 810 with the image processing device 700,which stores data onto the portable storage medium 825 or whichcommunicates with the image processing device 700 via the network 830.In this case, the device 500 could be connected to a camera 105 in orderto obtain the images 300.

In FIG. 8D, the devices 500, 700 are combined within a single device800. This single device could be integrated within or connected to acamera 105, allowing a user to preview images in real time as they arecaptured and, at the same, allowing the full-size images to be queuedand processed. In this embodiment, each of the devices 500, 700 areconnected to a storage medium 805. The device 500 generates full-sizeimages 300 and associated imaging effect parameter(s) for each image.The images and parameter(s) are stored in the storage medium 805. Thedevice 700 then performs an imaging effect on each of the full-sizeimages 300 using its associated imaging effect parameter(s). Thefull-size images, with the imaging effect applied, are then stored inthe storage medium. Using this embodiment, images can be captured andpreviewed with the imaging effect applied. The full-size images, withthe imaging effect applied, will then be available on the storage medium805 at a later time, once the device 700 has finished processing thefull-size images.

The one or more imaging effect parameters associated with each capturedimage 300 can take a variety of formats. For example, the parameter(s)associated with a particular image can be included in the image dataitself as additional metadata. When the image processing device 700receives the image 300, it extracts the metadata and can thus obtain theparameter(s) for performing the imaging effect. Alternatively, the imageprocessing device 500 can send the image data and parameter(s)associated with the image 300 as separate data. In this case, theparameter(s) for each image must be stored and recorded in such a waythat the imaging processing device 700 can look up the parameter(s) fora particular image during processing of that image. In this case, theparameter(s) for each image may be recorded in a look-up table, such asthat shown in FIG. 9.

The table of FIG. 9 comprises a first column 905 listing identifyingdata which uniquely identifies each of the captured images 300. Theidentifying data could be, for example, a unique filename for each ofthe images, such as a unique numerical filename. It also comprises asecond column 910 which lists an imaging effect parameter for each ofthe captured images 300. In this particular example, the shift parameterfor each image is an angle of rotation (in degrees) for correction ofthe image according to the shift correction method of FIG. 4.

A look-up table, such as that of FIG. 9, may be generated by the imageprocessing device 500 and transmitted to the image processing device700. This could be the case in the embodiment of FIG. 8B, for example,where the look-up table could be recorded to the portable storage medium825 along with the image data. It could also be the case in theembodiment of FIG. 8D, where the look-up table could be recorded to thestorage medium 805 along with the image data.

Alternatively, the table of FIG. 9 may be generated by the imageprocessing device 700 as it receives a stream of captured images 300 anda parameter for each image. For each image received, the imageidentifying data and the parameter are recorded in the table as theimage joins the queue for processing. This could be the case in theembodiments of FIG. 8A or 8C, for example, in which a constant stream ofimages and associated parameters may be transmitted from the device 500to the device 700. FIG. 10A illustrates a flow chart demonstrating theprocess carried out by the image processing device 500. The processstarts at step 1000. At step 1005, the device 500 receives an image. Atstep 1010, a smaller-sized representation of the image is generated. Atstep 1015, an imaging effect is performed on the smaller sizedrepresentation of the image. The imaging effect is performed on thebasis of an imaging effect parameter. An example of an imaging effect isa shift correction of the image. In this case, one example of theimaging effect parameter is the shift parameter 910 from FIG. 9. It isenvisaged that any kind of imaging effect may be performed. For example,the image may be made transparent. In this case, the imaging effectparameter will be the required degree of transparency. The imagingeffect parameter may therefore be seen as a parameter that defines thedegree of image effect to be applied to the image.

At step 1020, the smaller-sized representation of the image, with theimage effect applied, is output for display. At step 1025, it is decidedwhether or not a shutter button has been pressed. The shutter button maybe pressed at an instant decided by the user so as to capture a stillimage or to begin the capture of a video sequence. If it is decided thatthe shutter button has been pressed, then the process moves onto step1030, in which the original image and imaging effect parameter areoutput. The process then ends at step 1035. On the other hand, if it isdecided that the shutter button has not been pressed, then the processsimply ends at step 1035.

FIG. 10B illustrates a flow chart demonstrating the process carried outby the image processing device 700. The process starts at step 1040. Atstep 1045, the original image and imaging effect parameter are received.At step 1050, an imaging effect is performed on the original image onthe basis of the imaging effect parameter. An example of an imagingeffect is shift correction of the image. At step 1055, the originalimage, with the imaging effect applied, is output. The process then endsat step 1060. Although the embodiments described so far perform a shiftcorrection on the image 300 to correct for the convergence of verticallines, it will be appreciated that shift correction for the convergenceof horizontal lines could also be performed on the image 300. In thiscase, an imaging effect parameter for a captured image could be thevalue of an angle by which the image is rotated to correct forhorizontal line convergence. The image 300 would, in this case, berotated about an axis which is perpendicular to the axis 400 illustratedin FIG. 4. In embodiments, both vertical line shift correction andhorizontal line shift correction may be performed on a single image.

Furthermore, as noted above, the present disclosure is not limited toshift correction as being the imaging effect that is applied to animage. Rather, any imaging effect, where it would be useful for a userto be able to preview the image effect in real time, may be applied bythe image processing devices 500, 700. In the case of a differentimaging effect being applied, the image pre-processor 520 of the device500 would be configured to perform this different imaging effect insteadof the shift correction as previously described. Similarly, the shiftcorrection controller 530 of the device 500 would be replaced with acontroller for generating imaging effect parameters relevant for thedifferent imaging effect instead of generating shift correctionparameters. The image processor 710 of the device 700 would then beconfigured to perform the different imaging effect on the originalfull-size images 300 on the basis of the different imaging effectparameters.

Examples of possible alternative imaging effects that could be appliedusing the present invention include the tilting of an image (digitallyreplicating the effect that can be achieved using a tilt lens), theoverlaying of virtual objects on the image, the application of warpingeffects to the image, the editing of the colour or contrast of the imageor the compensation for barrel or pincushion lens effects, vignetting orshading of the sky. Of course, these merely serve as examples and do notconsitute limitations of the present invention.

FIG. 11 shows a live camera shoot 1100. A camera 1110 in someembodiments is mounted on a tripod 1112. The tripod 1112 includes a tiltsensor (not shown) such as an accelerometer or gravimeter. The tiltsensor measures the amount of tilt (pitch) applied to the camera 1110.The tripod 1112 also includes a pan sensor (not shown) that measures thedegree of panning (yaw) applied to the mounted camera. Roll in thisexample is assumed to be zero, with the tripod 1112 restricting themovement of the camera 1110 so that it cannot roll However, if thetripod 1112 is not set to prevent the roll of the camera, then thetripod 1112 could also include a roll sensor (not shown) that measuresthe degree of roll of the camera 1110. Typically, the pan sensormeasures the degree of pan away from a centre line.

The camera 1110 is controlled by a camera operator 1115. In embodimentsthe camera operator 1115 is positioned close to the camera 1110, thecamera operator 1115 may be located remotely from the camera 1110. Thecamera operator 1115 controls the operation of the camera 1110 bycontrolling the focus, zoom and position of the camera 1110.

A director of photography 1120 is also shown in FIG. 11. The director ofphotography is responsible for capturing the scene in a live camerashoot. The director of photography 1120 uses a tablet type computer 1600such as a Sony® Xperia® Tablet. For security, the tablet computer 1600asks the director of photography to identify himself or herself on thetablet and enter a password associated with the user. The tablet typecomputer is one example of a processing device. Other such devicesinclude (but are not limited to) computers such as laptop computers,netbooks or smartphones such as Sony® Xperia® Z. The processing deviceaccording to embodiments is shown in more detail with reference to FIG.16.

The tablet 1600 connects to the camera 1110 using a wireless connection.In particular, the tablet 1600 connects using the Digital Living NetworkAlliance® (DLNA) wireless standard. This enables the images captured bythe camera 1110 to be transferred to the tablet 1600. The application ofimage processing, in embodiments, is then performed using the tablet1600 before being displayed to the director of photography for review.Of course, the processing of the image captured using the camera 1110may be performed in the camera 1110. Alternatively, the images may bepartially processed in the camera 1110 with the remainder of theprocessing being provided in the tablet 1600. Indeed, as will beexplained later in respect of FIG. 16, the tablet 1600 may be used todisplay the captured image prior to processing.

The live camera shoot 1100 involves the camera 1110 shooting a liveaction pan of a building 1105. As will be apparent from FIG. 11, thebuilding 1105 is much taller than the camera 1110 and both the cameraoperator 1115 and the director of photography 1120.

FIG. 12 shows the pan of camera 1110. The camera 1110 is mounted ontripod 1112. Initially, camera 1110 is placed in position A. That is,the camera 110A is positioned at an angle θ₁ relative to a lineperpendicular to the building 1105. This is shown as 1110A is FIG. 12.The mounted camera 1110 then pans across the building to position C.When in position C, the camera 1110C is positioned at an angle θ₂relative to the line perpendicular to the building 1105. As will beapparent, during the pan from position A to position C the camera 1110moves through an angle θ₁+θ₂.

When in position B, the camera 1110B directly faces the building 1105.In other words, when in position B, the camera 1110 faces the lineperpendicular to the building 1105.

The panning motion between position A and position C may be smooth. Thatis, the angular velocity of the pan is constant for the entire panbetween position A and position C. Alternatively, the angular velocityof the pan may vary between position A and position C. For example, theangular velocity of the pan between position A and position B may behigher than the angular velocity of the pan between position B andposition C. Indeed, any speed in panning motion is envisaged.

FIG. 13 shows an array 1300 of the field of view of the camera whenlocated in position A, position B and position C respectively.Specifically, when located in position A, the field of view of camera1110 is shown in 1300A. Moreover, when located in position B, the fieldof view of camera 1110 is shown in 1300B and when located in position C,the field of view of camera 1110 is shown in 1300C.

Referring to the field of view in position A 1300A, an edge of thebuilding is shown. As the camera 1110 is positioned lower than thebuilding, but is directed to the top of the building, the image plane ofthe camera 1110 is not parallel to the building. This results in anincorrect perspective. Specifically, the captured edges within thebuilding are sloped rather than straight. As exemplified in FIG. 13, thewalls and the windows of the building appear sloped. This incorrectperspective within a still image can be corrected by performing a shiftcorrection using Adobe® Photoshop®. However, as noted hereinbefore, sucha correction may crop the image undesirably.

FIG. 14A shows a more detailed representation of image 1300A from FIG.13. As is indicated in FIG. 14A, the wall of the building is an angle ofφ_(V) from a straight vertical line 1310 and the roof is sloping at anangle of φ_(H) from a straight horizontal line 1320. For ease ofreference, the straight vertical line 1310 and the straight horizontalline 1320 are denoted by dashed lines in FIG. 14A. It should be notedhere that the wall of the building is one example of a straight verticaledge and the roof of the building is one example of a straighthorizontal edge. Other objects within the image include windows whichhave both straight vertical edges and straight horizontal edges. As willbe explained later in respect of FIGS. 15A-15B, in embodiments of thedisclosure, by determining φ_(V) and/or φ_(H) for edges in an image, itis possible to automatically determine which sloping edges should bevertical and/or which sloping edges should be horizontal, and to performshift correction on the image accordingly.

Before discussing how shift correction may be performed in embodimentsof the present invention, a number of considerations which must be takeninto account when performing shift correction will be discussed withreference to FIGS. 14B and 14C.

FIG. 14B shows a larger version of image 1300B from FIG. 13. It can beseen that vertical line convergence occurs in the image, with the edges1310, 1315 of the building 1105 appearing closer together nearer the topof the image. The edge 1310 is defined between positions (x₀, y₀) and(x₂, y₂) and the edge 1315 is defined between positions (x₁, y₁) and(x₃, y₃). The angles of the edges 1310, 1315 with respect to thevertical at each of the positions are also shown. Specifically, edge1310 has vertical angles φ_(VLU) and φ_(VLL) at positions (x₀, y₀) and(x₂, y₂), respectively, and edge 1315 has vertical angles φ_(VRU) andφ_(VRL) at portions (x₁, y₁) and (x₃, y₃), respectively. Also shown inthe image is the image centre line 1320. In the image 1300B, verticalline convergence will be more extreme at the edges of the image than atthe centre. This is a general characteristic of the perspectivedistortion which occurs due to an image sensor not being parallel to theplane of an object in the scene, as discussed previously. In anidealised scenario, this means that there will be zero vertical lineconvergence along the centre line 1320 of the image. Then, moving awayfrom the centre line, the vertical line convergence will increase. Thisresults in the non-zero angles φ_(VLU) and φ_(VLL) of the edge 1310 andφ_(VRU) and φ_(VRL) of the edge 1315. It is also noted that, since thebuilding 1105 is not completely central to the image 1300B, the edge1315 is slightly further from the image centre line 1320 than the edge1310. Hence, the angles φ_(VRU) and φ_(VRL) of the edge 1315 will beslightly larger than the angles φ_(VLU) and φ_(VLL) of the edge 1310.

The fact that different portions of the image are susceptible todifferent levels of distortion may need to be taken into account whenperforming shift correction on the image. More specifically, if an edge(or line) in the image is used as a reference for performing shiftcorrection on the image (as will be explained later), then the positionof the edge with respect to the centre of the image may be important indetermining whether a particular edge is suitable to be used as such areference.

This is illustrated further with respect to FIG. 14C, which shows alarger version of image 1300A in FIG. 13. In image 1300A, an image ofthe building 1105 has been captured from a different perspective. Theedges 1310, 1315 of the building 1105 thus appear in different positionsin the image frame. Specifically, edge 1310 is now defined betweenpositions (x₀′,y₀′) and (x₂′, y₂′) and edge 1315 is now defined betweena position (x₁′, y₁′) and a further position which is now outside theframe of the captured image. The vertical angles are also different.Specifically, edge 1310 now has vertical angles φ_(VLU)′ and φ_(VLL)′and the remaining end-point of edge 1315 now has vertical angleφ_(VRU)′. Notably, edge 1310 is now aligned with the centre line 1320,and hence the vertical angles of edge 1310, φ_(VLU)′ and φ_(VLL)′, areeffectively zero. On the other hand, edge 1315 is now further from thecentre line compared to image 1300B, and hence the vertical angleφ_(VRU)′ is larger than the original vertical angle φ_(VRU). As will beexplained later, shift correction may be performed on an image byrotating the image in 3D space (as shown in FIG. 4) until apredetermined reference edge in the image is aligned with the truevertical. In image 1300A, edge 1310 would not be suitable as a referencefor performing shift correction, since it is essentially alreadyvertical. On the other hand, edge 1315 would be much more suitable,since it exhibits substantial vertical line convergence distortion.

As will be apparent, embodiments of the present invention help to ensurethat shift correction is performed using reference edges in a suitableposition of a captured image.

Another consideration for shift correction exemplified by FIGS. 14B and14C is that objects for which line convergence distortion is a problemmay vary considerably in size in different images. In FIG. 14B, forexample, the perspective of the camera makes the building 1105 appearlarge. In other words, the building features in a large proportion ofthe area of the image 1300B. On the other hand, in FIG. 14C, theperspective of the camera makes the same building 1105 appear smaller.In other words, the building features in a significantly smallerproportion of the area of the image 1300A. Although it is unlikely to bea problem in this case (since in both images, the building 1105 appearsas a relatively large object), it is important that reference edges forperforming shift correction are chosen from objects which represent asufficiently large area of the image, so that an image is notincorrectly shift corrected based on unimportant image features (forexample, small objects which nonetheless have prominent linear features,such as above ground power-lines).

As will be apparent, embodiments of the present invention help to ensurethat shift correction is only performed using reference edges fromobjects which feature significantly in a captured image. FIGS. 15A-15Bshow a flow chart 1500 explaining process of automatically determiningwhich edges should be corrected. The process starts at step 1505. Animage is captured by an image sensor in the camera. The image isprovided to the processing device 1600. Therefore, the image is receivedby the processing device 1600 in step 1510. The received image is thenanalysed to detect the straight lines in the received image in step1515. This analysis may be performed using a known technique such asCanny edge detection, Marr-Hildreth edge detection or a Hough transform.Indeed, any type of edge detection may be performed.

The pixel co-ordinate of the start and end of each line is thenidentified in step 1520. In any one image, a large number of lines maybe detected. In order to reduce the processing burden, only lines deemedof significance in the image will be processed further. A line is deemedof significance, in embodiments, if the line exceeds a predeterminedlength (step 1525). This length may be a specified number of pixels inlength or a certain percentage length of the screen. However, otherfactors may additionally or alternatively deem a line to be ofsignificance. For example, lines further from the centre of the screenmay be deemed more significant than those nearer to the edge of thescreen. This is because the convergence of parallel lines tends to bemore extreme at the periphery of the screen compared to the centre.Therefore, a weighting may be applied to the line depending on itsproximity to the centre of the screen, so that lines which are locatedcloser to the periphery of the screen may be deemed significant even ifthey are shorter.

Returning to FIG. 15A, if the line does not exceed a threshold, the lineis ignored (step 1530). If the line does exceed a threshold and so theline is deemed significant, a flag is applied to the line and it isdetermined whether there are any more lines to test (step 1535).

If all lines detected in step 1515 have been tested, the algorithm movesto step 1537. On the other hand, if not all lines detected in step 1515have been tested, the algorithm moves to step 1525.

In step 1537, it is determined whether it is a shift correction tocorrect for vertical line convergence (also known as verticalcorrection) or a shift correction to correct for horizontal lineconvergence (also known as horizontal correction) that is required. Thisis something which can be determined by the user prior to the start ofthe process 1500. The user will be aware of the type line convergencethat is likely to be encountered in the scene, and can therefore set theprocess to handle vertical or horizontal correction as appropriate. Forexample, in a scene where there are a number of tall buildings, verticalline convergence is likely to be an issue, and thus the user can selectthat vertical correction is required. In step 1540, each line thatexceeds the threshold length is analysed to determine the value of φ_(H)(for the case of horizontal correction) or φ_(V) (for the case ofvertical correction). In order to determine φ_(H) or φ_(V), the pixelposition in the image of both end points on the line is used.

Referring to FIG. 15B, a line 1590 is shown. The line 1590 connects afirst pixel 1585 and a second pixel 1580. The first pixel 1585 islocated at pixel position (x1, y1). The second pixel 1580 is located atpixel position (x2, y2). Using trigonometry, it is shown that

$\varphi_{H} = {\tan^{- 1}\frac{\left( {{y_{1} - y_{2}}} \right)}{\left( {{x_{1} - x_{2}}} \right)}}$$\varphi_{V} = {\tan^{- 1}\frac{\left( {{x_{1} - x_{2}}} \right)}{\left( {{y_{1} - y_{2}}} \right)}}$

Therefore, it is possible to calculate φ_(H) or φ_(V) for eachsignificant line.

Returning to FIG. 15A, after the value of φ_(H) or φ_(V) has beencalculated for each significant line, each value of φ_(H) or φ_(V) iscompared to a threshold value and those lines having a φ_(H) or φ_(V)lower than a threshold are identified. This is step 1545. The purpose ofthe threshold value is to determine which significant line is affectedby the image plane of the camera 1110 not being parallel to the building1105 and so should be a straight horizontal or vertical line in a shiftcorrected image. For example, significant lines that have a φ_(H) orφ_(V) of less than 20° for example will be identified as lines thatshould be straight horizontal lines or straight vertical linesrespectively. Of course, other threshold values may be selected. In step1550, the lines having a value of φ_(H) or φ_(V) below the thresholdvalue are then analysed. A subset of the lines is selected. The subsetmay contain one or more lines. The one or more lines within the subsetmay be selected in any suitable way. For example, a certain number ofthe longest lines (for example, the five longest lines) may be selected.Alternatively, one or more lines may be selected on the basis of theirpositions with respect to the centre of the image, with lines which arefurther from the centre (and which therefore experience more extremeline convergence) being favoured above lines closer to the centre. Ofcourse, the one or more lines in the subset may be selected on acombination of both the positions and the lengths of the lines. Ingeneral terms, the one or more lines in the subset are chosen to be thelines which are most susceptible to line convergence and which aretherefore most suitable as references for performing shift correction onthe image.

In step 1552, it is then determined whether or not there is more thanone line in the subset. If there is not more than one line, then theprocess moves to step 1553, in which the user is warned that there maybe insufficient edge information in the image to perform an automaticshift correction. The process then moves to step 1554, in which inputfrom the user is awaited as to whether or not the process shouldcontinue. If the user accepts the risk that there may be insufficientedge information, but still wants the automatic shift correction to beperformed, then the process continues and moves to step 1565. On theother hand, if, based on the fact that there may be insufficient edgeinformation, the user decides that the automatic shift correction shouldnot be performed, then the process ends at step 1575.

Advantageously, by warning the user that the capability of the automaticshift correction may be hindered by a lack of suitable information inthe image (as occurs when there is only one line to refer to in theimage), the user is able to more easily monitor quality control of theprocessed images.

If, at step 1552, it is determined that there is more than one line inthe subset, then the process moves on to step 1555. In step 1555, anarea of a plane defined by the end-points of the lines in the subset isdetermined. The area is then compared to a threshold. For example, itcould be determined as to whether or not the area defined by theend-points of the lines covers a certain proportion of the whole image(for example, 20% of the image). If it is determined that the area isless than the threshold, then the process moves to step 1553, in whichthe user is warned that there may be insufficient edge information inthe image to perform an automatic shift correction. The process thenmoves to step 1554, in which the user may choose to continue with theautomatic shift correction (step 1565) or to end the process (step1575), as previously described. On the other hand, if it is determinedthat the area is greater than or equal to the threshold, the processmoves onto step 1565.

Advantageously, by warning the user that the capability of the automaticshift correction may be hindered by a lack of suitable information inthe image (as occurs when the lines in the subset only relate to a smallpart of the image), the user is able to more easily monitor qualitycontrol of the processed images. In step 1565, shift correction is thenperformed on the image on the basis of the one or more lines in thesubset. This transformation can be performed using any suitable method.

In a first example, a most significant line is chosen from the subset.The most significant line could be chosen using any suitable criteria.One criteria is that the line in the subset which is furthest from thecentre of the image (for example, the line which has its centre pointfurthest from the centre of the image), and is thus most susceptible toline convergence, is selected as the most significant line.Advantageously, this allows the shift correction of the image to beperformed so as to correct for the most extreme line convergence whichis likely to occur in the image.

The image is then shift corrected by transforming the image so that themost significant line becomes either a straight horizontal line (in thecase of horizontal correction) or a straight vertical line (in the caseof vertical correction). The image will be transformed by rotating theimage in 3D space in a similar way to that shown in FIG. 4 until themost significant line becomes a straight horizontal or vertical line.That is, the image will be rotated until the horizontal φ_(H) orvertical φ_(V) angle of the most significant line is minimised or ismade to be below a threshold. This ensures that the entire image isshift corrected relative to the most significant line.

In a second example, a plurality of lines in the subset could be used asa reference when performing the shift correction. In this case, theimage is rotated in 3D space until an average value of φ_(H) or φ_(V) ofthe plurality of lines is minimised or is made to be below a threshold.Advantageously, this allows the shift correction to be optimised formore than one converging line in the image.

In both the first and second shift correction examples discussed above,the most significant line (first example) or average line (secondexample) value of φ_(H) or φ_(V) can be minimised by incrementallyrotating the image in 3D space between two predetermined rotationlimits. For example, the image could be rotated between ±20° inincrements of a fixed incrementation angle, such as an angle of 0.2°.For each incremental rotation, the most significant line or average linevalue of φ_(H) or φ_(V) could be determined. The image would then bedetermined to be shift corrected for the incremental rotation for whichthe most significant line or average line value of φ_(H) or φ_(V) isminimised. The total amount of 3D rotation of the image in order for itto be deemed shift corrected is called the shift correctiontransformation, and may be denoted by an angle φ_(H) (for horizontalcorrection) or φ_(V) (for vertical correction). The shift correctiontransformation is an imaging effect parameter.

It is noted that in all shift correction methods that involve rotatingthe image in 3D space, it is also necessary to scale of the image tocorrect for distortional effects that may result from the imagerotation. For horizontal correction, the image may be scaledhorizontally by cos (ρ_(H)) to correct for horizontal distortionaleffects. On the other hand, for vertical correction, the image may bescaled vertically by cos (ρ_(V)) to correct for vertical distortionaleffects.

The process of automatically analysing the image and performing shiftcorrection ends at step 1575. It is noted that if an image requires bothhorizontal and vertical correction, then the process 1500 may be appliedtwice, once for horizontal correction and once for vertical correction.In this case, an image will have both a horizontal shift correctiontransformation ρ_(H) and a vertical shift correction transformationρ_(V). The shift correction transformation ρ_(H) and/or ρ_(V) which isused to correct the image is an imaging effect parameter which may bestored as metadata in association with an identifier for the frame ofvideo. This will be explained later with reference to table 1. Thisallows any further post-processing to be carried out on the image shouldfurther processing be required. Also, if during post-processing it isdecided not to apply the shift correction, the correction applied may bereversed using the stored value of ρ_(H) or ρ_(V) to revert to theoriginal, un-corrected, image.

An embodiment of the tablet 1600 is shown in FIG. 16. The tablet 1600has a display 1630 upon which the image is displayed. It is important tonote that the image displayed to the user may be a corrected image or anuncorrected image. In the case of FIG. 16, the image is uncorrected.Also displayed in a corner of the display 1630 is a mode selection box1605. The mode selection box 1605 is located in a corner of the display1630 to not obscure the image received at the tablet 1600. Clearly, thesize and location of the mode selection box 1605 may vary depending uponthe image received.

In the mode selection box 1605 a user may choose between a manual modeand an automatic mode of operation. The selection is made by the user bypressing either the manual mode radio button 1610 or the automatic moderadio button 1615.

If the automatic mode radio button 1615 is pressed, the process definedin FIGS. 15A and 15B is performed. However, if the manual mode radiobutton 1610 is pressed, the displayed image is not corrected. Instead,the lines within the image are detected by a known technique such as aCanny Edge Detection, Marr-Hildreth Edge detection or a Hough transformor the like. The user may then touch a line in the image. This selectsthat line. In FIG. 16, the user 1625 touches one edge of the building.This edge is then highlighted by area 1620. The highlighting may includea changing the colour of the line, applying an overlaid line orhighlighting the start and end point of the line. If the user 1625touches the selected line again, the user de-selects the line.

By selecting the line, the user 1625 indicates to the tablet 1600 thatthe selected line is both a significant line and that the line should behorizontal or vertical. The tablet 1600 transforms the image so that thechosen line becomes a straight horizontal or vertical line. The tablet1600 calculates whether the selected line should be a horizontalstraight line or a vertical straight line by determining which of φ_(H)and φ_(V) is lower. Specifically, if φ_(H) is lower than φ_(V), theimage should be transformed so that the line becomes a straighthorizontal line, and if φ_(V) is lower than φ_(V), the image should betransformed so that the line becomes a straight vertical line.

The selected line is then tracked as the camera pans. The tracking oflines and objects in a sequence of images is known and will not bediscussed further. At each image in the sequence of images, the value ofφ_(H) and φ_(V) is re-calculated to ensure that the selected straightline remains a straight horizontal line or straight vertical line. Thisis because the value of φ_(H) and φ_(V) may change as the camera pans.After calculation of φ_(H) and φ_(V) in each image, at each image in thesequence of images, the image is transformed so that the selected linebecomes a straight horizontal or vertical line.

In the description above, the line selected by the user is used as theonly reference for performing shift correction on the image.Advantageously, this allows the user to select a line which exhibits themost prominent line convergence distortion and to correct the image bytransforming it until the selected line becomes sufficiently horizontalor vertical.

However, in embodiments, it may be possible for the user to selectmultiple lines in the image to be used as a reference for performingshift correction on the image. The image could then be shift correctedeither by the tablet 1600 automatically determining the most significantline (for example, the line furthest from the centre of the image) andcorrecting the image with respect to this most significant line.Alternatively, the image could be shift corrected by minimising theaverage horizontal or vertical angle of the selected lines. Both ofthese methods have been discussed earlier with respect to FIGS. 15A-15B.In this case, when the manual mode radio button 1610 of the selectionbox 1605 is chosen, an additional selection box may appear (not shown)which allows the user to choose whether it is horizontal or verticalshift correction which is required.

Advantageously, if there is not a single prominent line which exhibitsline convergence distortion, this allows the user to select a number oflines which may be suitable references and to shift correct the imagebased on these lines.

In embodiments, it may also be possible for the user to select one ormore lines which correspond to one or more objects in the scene (forexample, different buildings). The tablet 1600 may recognise theseobjects using any suitable object recognition algorithm (these are wellknown in the art and are therefore not detailed here), and the user maythen select detected edges and associate each selected edge with one ofthe recognised objects. For example, the user could tap the screen ofthe tablet 1600 once to selected a detected edge (for example, an edgeof a building in the image), and then tap the screen of the tablet asecond time to select a recognised object in the image (for example, thebuilding itself). This would associated the selected edge with therecognised object. Then, as the camera is panned, the image correctioncan be performed on the basis of the edges which have been associatedwith chosen the objects, as long as those objects remain in the capturedimages.

Advantageously, this allows the same reference edges to be used forshift correction of successively captured images, rather than differentedges for different images (which may occur as the camera is panned andas successively captured images have different edges which may bedetected). This results in a more consistent level of shift correction.Also, the edges of objects which are most susceptible to lineconvergence distortion can be chosen and associated with those objects,resulting in consistent shift correction for those objects forsuccessively captured images.

FIG. 17 shows the tablet computer 1600 having the image shown in FIG. 16corrected relative to the selected line 1620. The corrected image isdisplayed on display 1630. As can be seen, the lines within thecorrected image have been transformed relative to the line selected inFIG. 16. This correction of the lines in the image relative to theselected significant line means that the perspective in the image iscorrected. Of course, it will be recognised that if multiple lines wereselected by the user for shift correction of the image, then the imagewould be corrected using these multiple lines, as already discussed.

A summary box 1700 is shown in FIG. 17. The summary box 1700 includesone or more metadata fields to be displayed to the user of the tablet1600. Specifically, the summary box 1700 is positioned on display 1630in the top left corner of the display. It should be noted that thesummary box 1700 may be located anywhere on the display 1630. It isadvantageous to place the summary box 1700 in an area of the display1630 in which no image is being displayed. This may require the positionof the summary box 1700 to move on the display 1630. So, in the exampleof FIG. 17, as the camera continues to pan across the building and theselected line 1620 moves to the left of the display 1630, the summarybox 1700 may be re-positioned to be displayed in the top right handcorner of the display 1630. In addition or alternatively, the size ofthe summary box 1700 may be increased or decreased in dependence uponthe use of the display 1630. So, instead of re-positioning the summarybox 1700 to the top right hand corner of the display 1630, the size ofthe summary box 1700 may remain in the same position but instead, bemade smaller. In order to reduce the size of the summary box 1700, thesize of font within the summary box 1700 may be reduced. Alternatively,or additionally, the number of fields within the summary box 1700 may bereduced. For example, in FIG. 17 a shot property field is shown.However, if this field is not of crucial importance to the user of thetablet 1600, this field may be the first field to be removed in theevent that the summary box 1700 needs to be made smaller. The user maytherefore prioritise the content provided in the summary box 1700. Thismay be performed during a set-up period explained later.

Within the summary box 1700 a good shot marker box 1705 may be ticked bythe user. The good shot marker box 1705 is a flag that is stored withinmetadata associated with the image. If the good shot marker box 1705 isselected, the currently displayed image is deemed a good shot and thegood shot marker flag is activated. This assists in searching forcontent that the user of the tablet 1600 considers to be good as thesearch only returns content having the good shot flag activated. If thegood shot marker is not selected by the user of the tablet 1600, thegood shot flag is not activated.

Also provided within the summary box 1700 is a shot properties field.The shot properties field includes information such as the take 1710, aunique identifier 1715 that uniquely identifies the frame currentlydisplayed and lens metadata 1720. The unique identifier may be globallyunique such as a unique material identifier (UMID), or may be unique inthe take or unique on the storage medium. The lens metadata 1720 mayinclude information such as focal length (indicating the level of zoomof the image), lens aperture or the like. The lens metadata 1720 is, inembodiments, provided automatically by the lens. Additionally, acomments box 1725 is provided within the summary box 1700. The commentsbox 1725 is a free-text field that allows the user of the tablet 1600 tomake observations against the received frame. If the user touches thecomments box 1725 a virtual keyboard is activated on the tablet 1600.The virtual keyboard enables the user to enter comments on the virtualkeyboard that are then stored in association with the image. Thecomments box 1725 may instead by a drop-down list of predefined commentssuch as “need hue enhancement”, “contrast too high” or the like. Byproviding the drop-down list, it is easier and quicker for the user ofthe tablet to associate predefined comments to the received frame. Thisis particularly useful when annotating moving images.

The comment made in the comments box 1725 is stored in association withthe received frame. The comment may be stored as metadata linked to thereceived frame using the unique identifier. A description of themetadata stored in association with the received frame is providedhereinafter in table A.

Additionally provided is a message box 1730. The message box 1730 istermed “Message to camera operator”. When selected by the user, avirtual keyboard is provided on the display 1630. The user of the tablet1600 then enters a message in the free-text message box. Similarly tothe comments box 1725, instead of a free-text box, a drop down menu ofpredefined messages may be provided. For example, a drop-down menu ofpopular messages such as “pan more slowly” or the like may be provided.This again is easier and quicker for the user of the tablet to associatepredefined comments to the received frame. Instead of the content of themessage box 1730 being stored in association with the received frame,the content of the message box 1730 is transferred over the wirelesslink to the camera 1100. The content of the message box 1730 isdisplayed to the camera operator 1115 through the view finder. Inaddition to the message, the identity of the user of the tablet (enteredwhen logging on to the tablet) is also displayed to the camera operator1115. This enables several tablet users to each send a message to thecamera operator 1110. The content of the message box 1730 may be storedin association with the unique identity or may be simply sent to thecamera 1110 and not recorded. This could enable real time or near realtime comments from a person reviewing or editing the captured content tobe sent to a camera operator.

As noted above, the metadata associated with each frame in a sequence offrames is associated with the frame. In the example above, each frame isallocated an identifier which uniquely identifies the frame. Themetadata is stored, in embodiments, in association with the frameidentifier. This metadata is stored separately to the frame.

Table 1 below shows the metadata associated with a frame.

TABLE 1 Good Shot Focal Length Unique ID ρ_(H) ρ_(V) Pan Tilt Marker(Zoom) Comment AC12:3ECF:23E2 X +1.2321° −32.873 +2.326 0 55.0 Contrasttoo low AC12:3ECF:23E3 X +1.2325° −15.324 +2.326 1 55.0 X AC12:3ECF:23E4X +1.2327° +10.654 +2.326 1 53.2 X AC12:3ECF:23E5 X +1.2324° +24.623+2.326 0 2.34251.7 X

In the column entitled “unique ID”, the unique identifier thatidentifies the frame is provided. In the column entitled “ρ_(H)”, thetransform required to perform shift correction in the horizontaldirection is stored. In this case, as the column contains “X”, thisindicates that no correction in the horizontal direction is performed.In the column entitled “ρ_(V)”, the transform required to correct thetilt-shift in the vertical direction is stored. In this case, the columncontains the vertical correction. The sign of the angle indicates thedirection of correction. In this example, the sign “+” indicates acorrection, whereas a sign “−” indicates a correction to the left.However, of course any nomenclature may be chosen. In the columnentitled “pan”, the value of pan of the camera 1110 when capturing theframe is stored. Similarly, in the column entitled “tilt”, the value oftilt of the camera 1110 when capturing the frame is stored. The value ofpan and tilt is provided by the tripod 1112.

It is emphasised that the angles ρ_(H) and ρ_(V) in Table 1 are the sameas the “Shift Parameter” angles in the table shown in FIG. 9.Specifically, ρ_(H) and ρ_(V) show an angle by which the image must berotated in 3D space in order for the image to be shift corrected.

In the column entitled “Good Shot Marker”, a binary indicationidentifies whether the user of the tablet 1600 has determined the shotis a good shot and has indicated this using the good shot box 1705. Inthe case of the frame being identified as a good shot, the column shows“1” and in the event that no good shot marker is associated with theframe, the column shows “0”. The column entitled “Focal Length” showsthe focal length of the camera capturing the image. The focal lengthindicates the amount of zoom or magnification of the captured image, andis important because different focal lengths of the camera lens (whichrepresent different zoom levels) result in different amounts of lineconvergence distortion. In particular, larger focal lengths/highmagnification levels result in reduced line convergence distortion andthus require a reduced level of shift correction. On the other hand,lower focal lengths/lower magnification levels result in increased lineconvergence distortion and thus require an increased level of shiftcorrection. In the column entitled “comments”, any comments entered inor selected from the comments box 1725 are stored.

Table 1 is stored within the tablet 1600 during capturing of the scene.After the capture of the scene has been completed, the metadata of table1 may be transferred over the wireless connection to the camera 1110 forstorage therein. This transfer may take place with the metadata beingtransferred separately to the images or together with the images.Alternatively, table 1 may be stored within the tablet 1600 fordistribution later.

Instead of storing the metadata separately to the frame, it is possibleto store the metadata within the frame. For example, the metadata may beembedded within the frame.

FIG. 18 shows a flow chart 1800 explaining operation of the camera andthe tablet.

The operation starts at step 1805. When powered on, the tablet 1600requests the user log on to the tablet 1600. In response to the log onin step 1810, the user's profile is retrieved from the memory of thetablet. This includes the user's name and settings. Also, the rightsassociated with the user, such as right to amend the categories in table1 are also retrieved during log-in. Further, as part of the log-onprocess, the tablet 1600 pairs with the camera 1110 using the DLNAstandard.

When filming, the camera operator presses record on the camera 1110.This indicates to the tablet 1600 that a stream of images will bereceived from the camera. This is step 1815. A first image is receivedand analysed by the tablet 1600. Additionally provided with the firstimage is the unique identifier which uniquely identifies the firstimage. The analysis is carried out to determine the lines within thefirst image. This is step 1820 and as explained with reference to FIGS.15A-15B, this is carried out using a Hough transform or the like. Indecision step 1825, the tablet 1600 determines whether the automaticbutton 1615 or the manual button 1605 in the selection box 1605 isselected. If the automatic button is selected, the value of φ_(H) orφ_(V) is determined for each significant line (depending on whether ornot horizontal or vertical correction has been selected by the user).This is step 1835. In step 1840, a subset of lines are selected andanalysed, according to the method described with respect to FIG. 15A.The subset of lines includes one or more lines.

On the other hand, if the manual button 1605 is selected, the tablet1600 waits to receive a manual selection of one or more lines in step1830.

The image is then transformed based on the one or more selected lines.This is step 1845. In other words, the shift correction is applied tothe whole image based on the one or more selected lines. As explained instep 1565 of FIG. 15B, the image is transformed so that a most suitableline is transformed to be a straight vertical or horizontal line, or sothat an average of the vertical or horizontal angles of each of aplurality of selected lines is minimised.

The metadata table (table 1) is accessed in step 1850. Specifically, themetadata table is accessed such that the columns are populated with theappropriate metadata. The metadata that is stored within table 1 isdetermined by the user of the tablet 1600 if that user has appropriateprivileges defined by the user profile stored within the tablet 1600.

Firstly, the unique identifier received with the first image is storedwithin table 1. The pan and tilt columns are then completed (step 1855).The pan and tilt information is generated by the tripod 1112. However,in embodiments, the pan and tilt information may be sent to the tablet1600 via the camera 1100. Alternatively, the tripod 1112 may communicatethis information directly to the tablet 1600. Furthermore, it isenvisaged that the pan and tilt information may be generated by thecamera 1100 itself by, for example, a calibrated accelerometer orgravimeter comprised within the camera. The pan and tilt information maythen be sent to the tablet 1600 directly from the camera 1100.

In step 1860, a decision is made to determine whether the user of thetablet 1600 has selected the first image as a good shot. In other words,a decision is made to determine whether the user has checked the goodshot marker box 1705. If the user of the tablet 1600 has selected thefirst image as being a good shot the “yes” path is followed and a “1” isapplied to the good shot marker column in table 1 (step 1870).Alternatively, if the good shot marker box is not selected, the “no”path is followed and a “0” is applied to the good shot marker column intable 1 (step 1865).

Next, in step 1875, the content of the comment box 1725 is checked. Ifthe user of the tablet 1600 has inserted a comment (either usingfree-text or a drop down menu), the comment is applied to the column intable 1 (step 1885). If the user of the tablet 1600 does not insert acomment, then an “X” is inserted to indicate no comment has been added(step 1880).

Finally, the tablet 1600 checks whether a message to the camera operatorhas been inserted in the message box 1730. If a message is inserted, themessage is transmitted, along with the identity of the user of thetablet 1600 to the camera 1110. This is step 1895. Alternatively, if nomessage is inserted, no message is transmitted to the camera 1110. Thetablet 1600 then checks whether filming has stopped. This is achievedbecause when the filming has stopped, a stop flag is transmitted fromthe camera 1110 to the tablet 1600. This check is carried out at step1897. If filming has stopped, the “yes” path is followed to the end step1899. Alternatively, if filming continues, the “no” path is followed andthe process returns to step 1815 where the second image in the sequenceof images is received.

The aforesaid description allows a director of photography or any userof the tablet 1600 to view the captured image with the correction orapplication of image processing at the time of filming. Therefore, ifuser of the tablet 1600 decides that the scene needs re-shooting, thenthis can be performed straightaway. In other words, the application ofimage processing, in embodiments, occurs at the time of image capturerather than during post production.

The aforesaid image processing takes place on every image. In otherembodiments, the image processing may take place on certain markedframes. This will now be explained with reference to FIG. 19. In image Aof FIG. 19, the camera 1110 is positioned at point A in FIG. 12. Inother words, the camera 1110 is panned to face the left-side of thebuilding. A tablet 1600A having the selection box 1700A and a markselection box 1910A is shown. The mark selection box 1910A is a togglebox on the touch screen of the display that the user presses to select.The selection by the user is indicated by a check appearing in the markselection box 1910A. By selecting the mark selection box 1910A, theimage correction explained in FIG. 18 is carried out. Additionally,Table 1 is completed for the marked frame. If the mark selection box1910A remains unchecked, however, the image processing explained withreference to FIG. 18 is not carried out. Instead an alternative imageprocessing, explained with reference to FIG. 20 is carried out.

In image B of FIG. 19, the camera 1110 is positioned at point B in FIG.12. In other words, the camera 1110 is panned to face the front of thebuilding. A tablet 1600B having the selection box 1700B and the markselection box 1910B is shown. Similar, in image C of FIG. 19, the camerais positioned at point C in FIG. 12. In other words, the camera 1110 ispanned to face the right side of the building. A tablet 1600C having theselection box 1700C and the mark selection box 1910C is shown.

As in the case of image A, because the mark selection box 1910B and markselection box 1910C in images B and C respectively are marked, the imageis corrected and the appropriate metadata is written to the table. Table2 shows a table explaining the marked images according to thealternative image processing.

TABLE 2 Good Focal Shot Length Com- Image ρ_(H) ρ_(V) Pan Tilt Marker(Zoom) ment A +1.342 X −45.000° +2.326 0 55.0 Contrast too low B +0.829X −0.002° +2.326 1 53.2 X C +1.286 X +40.654 +2.326 1 51.7 X

In FIG. 20, a graph of ρ_(H) against pan is shown. This graph is formedfrom the metadata stored in table 2. Specifically, the graph 2000 showsthe value of ρ_(H) for image A (point 2010A), image B (point 2010B) andimage C (point 2010C). The relationship between point 2010A and point2010B is linear (line 2020A) and the relationship between point 2010Band point 2010C is linear (line 2020B). Of course, although a linearrelationship between the points is shown, in embodiments, otherrelationships between the points are envisaged such as exponential,polynomial or squared type relationships.

In order to determine the value of ρ_(H1) for the image captured whenthe camera 1110 is at a pan position of −10.232° (2030A), the value ofρ_(H1) is obtained from line 2020A. Further, in order to determine thevalue of ρ_(H2) for the image captured when the camera 1110 is at a panposition of 39.231° (2030B), the value of ρ_(H2) is obtained from line2020B.

In use, therefore, the camera 1110 may move across a scene during arehearsal. During the rehearsal, the director of photography (who, inone example, is the user of the tablet 1600) marks several positionsduring the rehearsal. The metadata associated with the position of thecamera (the pan and/or tilt) of the camera 1110 is captured from thecamera or the tripod 1112. The value of ρ_(H) and/or ρ_(V) is calculatedby the tablet 1600 and stored in association with the position of thecamera 1110 when the image is marked. During live capture, the camera1110 follows the same path and the tablet 1600 then interpolates thevalue of ρ_(H) and/or ρ_(V) for any value of pan and/or tilt from themarked positions. Therefore, the tablet 1600 will only need to calculatethe value of ρ_(H) and/or ρ_(V) for marked images once during rehearsaland for all other images in the sequence captured during live capture,the tablet 1600 will interpolate the value of ρ_(H) and/or ρ_(V). Thisefficiently uses processing power within the tablet 1600 compared withcalculating the value of ρ_(H) and/or ρ_(V) for each image.

The interpolation technique discussed with reference to FIG. 20 occurson the assumption that the focal length of the lens changes betweenpositions A, B and C during live capture in the same way as it changedduring the rehearsal (that is, from 55.0 mm in position A to 53.2 mm inposition B, and from 53.2 mm in position B to 51.7 mm in position C).This is important since, as already discussed, different focal lengthsof the camera lens (representing different zoom levels) result indifferent amounts of line convergence distortion and thus requiredifferent shift correction transformations ρ_(H) and/or ρ_(V). Theinterpolated shift correction transformation may not be correct ifdifferent focal lengths to those used in the rehearsal camera pan areused.

In embodiments, if, during live capture, the camera reaches one of thepredetermined positions A, B or C with a different focal length to thatrecorded in the metadata in table 2 during rehearsal for that position(allowing for an appropriate margin of error, such as 5%), then awarning is issued to the user. In addition, the focal length error (thatis, the difference between the focal length for the position recordedduring rehearsal and the focal length for the position recorded duringlive capture, when the difference exceeds the acceptable error margin)may be recorded as additional metadata. The focal length error may thenbe used for correction of the interpolated shift corrected imagesobtained during live capture, either in real time via the tablet 1600,or later during post-processing.

In embodiments, the correction of the interpolated shift correctedimages using the focal length error may take place using the methodillustrated in FIG. 21. FIG. 21 shows an enlarged portion of the graphof FIG. 20 with some extra data. Specifically, FIG. 21 shows a graph ofthe interpolated shift correction transformation ρ_(H) against thecamera pan as the camera moves from position A to position B.

A first dashed line 2020A is the same line as that found in FIG. 20, andindicates the interpolated shift correction when the zoom of the cameralens in positions A and B is the same as that used during the rehearsal.Here, the interpolated shift correction transformation at camera panposition 2030A is ρ_(H1). However, FIG. 21 also shows a second dashedline 2020A′ which is not shown in FIG. 20. Line 2020A′ indicates theinterpolated shift correction when the zoom of the camera lens inposition B is different to that used during the rehearsal. In this case,a lower focal length at position B compared to that used during therehearsal has been used, resulting in a lower magnification and moreprominent line convergence distortion. From the focal length error (thatis, the difference between the rehearsal focal length at position B andthe different, lower focal length used in live capture at position B,which will be negative in this case), the tablet 1600 recognises that alarger shift correction transformation ρ_(H) will be required. Thelarger shift correction transformation ρ_(H) results in the point2010B′. The tablet may calculate the larger shift correctiontransformation ρ_(H) from the focal length error (which represents arelative change in magnification) using a lookup table associated withlens being used with the camera 1110. The lookup table may be sent fromthe camera 1110 to the tablet 1600 as additional metadata. Theinterpolated shift correction transformation ρ_(H) may thus becalculated using the new dashed line 2020A′ which extends from 2010A to2010B′. So, the interpolated shift correction transformation at camerapan position 2030A becomes ρ_(H1)′ instead of ρ_(H1).

Advantageously, the interpolated shift correction transformation foreach image captured as the camera moves from position A to position B isthus corrected to take into account changes in the focal length of thecamera lens with respect to the rehearsal. This maintains the efficientprocessing of only calculating the shift correction for marked images(instead of all images), but also gives more flexibility to the cameraoperator to change the focal length/zoom level during live capture.

In the embodiments of FIGS. 19-21, it is noted that, rather than usingan edge detection technique, the shift correction transformation ρ_(H)and/or ρ_(V) for each of the marked images captured at the positions A,B and C could be entered manually by the user. For example, for eachmarked image, the user could enter a numerical value of ρ_(H) and/orρ_(V) using a numerical keypad displayed on the tablet (not shown) orcould gradually increase or decrease the value of ρ_(H) and/or ρ_(V)using increase and decrease buttons displayed on the tablet (not shown)until the line convergence distortion is observed to be satisfactorilyremoved. These manually entered values of ρ_(H) and/or ρ_(V) could thenbe saved as the metadata in table 2 and used for the interpolationtechniques discussed above.

FIG. 22 shows a block diagram of the tablet 1600. The tablet 1600 iscontrolled by a processor 1810. The processor 1810 is controlled bycomputer readable instructions, which when run by the processor 1810,makes the tablet 1600 perform certain steps. Memory 1815 is connected tothe processor 1810. The memory 1815 stores a computer program whichcontains the computer readable instructions which, when loaded onto theprocessor 1810 controls the processor 1810. The memory 1815 may be anykind of memory such as solid state memory, optical readable memory,magnetic readable memory or the like. Additionally stored within thememory 1815 is the metadata in table 1 and/or table 2. The metadata maybe output from the memory separately to the image stored within thememory 1815. The display 1630 is also connected to the processor 1810and is controlled thereby. Overlaid on the display 1630 is a transparenttouch screen 1830. By pressing the touch screen 1830, the user interactswith the tablet 1600. A DLNA processor 1825 is connected to theprocessor 1810. The DLNA processor 1825 communicates with the camera1110 to form a wireless link therewith. Finally, the tablet 1600includes a speaker 1820 for outputting sound. The speaker 1820 isconnected to the processor 1810.

FIG. 23 shows a block diagram of the camera 1110 and the tripod 1112.The camera 1110 comprises a camera processor 2100 which controls theoperation of the camera 1110. Connected to the camera processor 2100 isa lens 2125, a Global Positioning System block 2110, camera storage2105, a DLNA block 2115, a sensor 2120 and a mounting connector 2130.The lens 2125 focuses the light onto an image sensor (not shown). Thelens 2125 also provides the lens metadata stored in table 1 and 2 whichincludes the focal length of the lens and other metadata associated withthe lens like the model of the lens and the like. The global positioningblock 2110 determines the geographical position of the camera 1110. Thesensor 2120 in the camera 1110 determines the angle of tilt and/or panof the camera. However, as will be appreciated, this may not be requiredif the angle of tilt and/or pan of the camera 1110 is determined by thetripod 1112, as is the case in this embodiment.

The DLNA block 2115 pairs the camera 1110 to the tablet 1600 using theDLNA standard so that data such as metadata and image data may be passedbetween the camera 1110 and the tablet 1600.

The camera storage 2105 stores images captured by the camera 1110 andmetadata associated with the lens 2125. Further, the camera storage 2105contains computer readable instructions which, when loaded onto thecamera processor 2100, configures the camera processor 2100 to performcertain methods consistent with embodiments of the disclosure.

The mounting connector 2130 contains a first part 2135A which, whenmounted onto the tripod 1112 engages with a second part 2135B. Thesecond part 2135B is coupled to a tilt and/or pan sensor 2145 thatdetermines the tilt and/or pan of the camera 1110 with reference to thetilt and/or pan of the tripod 1112. The tilt and/or pan sensor 2145communicates with the camera 1110 via the engaged first part 2135A andthe second part 2135B of the mounting connector 2130. The tilt and/orpan of the tripod is controlled by the grip 2140 and the height of thetripod is adjusted by adjusting one or more legs 2150 of the tripod1112.

It should be noted that although the foregoing relates to images, theseimages may be full resolution images such as so-called High Definitionimages, or even Super High Definition images such as 4K or even 8Kimages. Alternatively, these images may be lower resolutionrepresentations of the images captured by the camera. For example, theimages used in the above embodiment may be preview images similar or thesame as those preview images described in earlier embodiments.

Further, in embodiments, the tablet 1600 may not perform thetransformation thereon. For example, the value of φ_(H) and φ_(V) may becalculated on the tablet 1600 and these values may be transferred to aseparate device for performing the image processing on the image. Thisseparate device may include computers located on the cloud for exampleas in the case of earlier embodiments. Alternatively, the device may bea stand-alone computer with specialised graphics processors. In otherwords, the calculation of the transformations may be calculated on thetablet 1600, but the intensive image processing may be carried out on aspecialist device that is located separately to the tablet 1600.

It will be appreciated that features of embodiments of this disclosurecould be combined as appropriate so as to benefit from the advantageousfeatures the disclosure.

For example, the image processing device 500 could be comprised withinthe tablet 1600. This would allow shift correction to be applied tosmaller-sized, preview images by the tablet using the line detectionmethod described with reference to FIGS. 15-16.

Specifically, an image from the camera 1100 could be transmitted to thetablet 1600 and received by the receiver 505 of the image processingdevice 500 comprised within the tablet 1600. The image could then beconverted to a smaller-sized, preview image by the converter 510,processed to apply the shift correction as an imaging effect by theimage pre-processor 520 and output by the display output 525 to thedisplay 1630 of the tablet 1600.

In this case, the shift correction applied by the image pre-processor520 would be achieved using the line detection method described withreference to FIGS. 15-16. Specifically, the values of φ_(H) and φ_(V)for the most significant line (the most significant line being detectedmanually or automatically) would be calculated by the shift correctioncontroller 530 by analysing the pre-view image generated by theconverter 510. The value of φ_(H) (in the case of a horizontal line) orφ_(V) (in the case of a vertical line) would then be output as animaging effect parameter to the image pre-processor 520 in order for theimage pre-processor to perform the shift correction. The imagepre-processor 520 would then perform the shift correction by rotatingthe image in 3D space until the value of φ_(H) or φ_(V) became zero (oras close to zero as possible).

The value of φ_(H) or φ_(V) would also be output to the image/parameteroutput 515 for storage and/or transmission. For example, the value ofφ_(H) or φ_(V) could be stored together with the values of φ_(H) orφ_(V) for other captured images of the scene in a table such as table 1(along with the other parameters, such as the pan and tilt information,received from the camera 1100). Once filming is complete, the tablecould then be transmitted back to the camera 1100 or another suitabledevice to be used in conjunction with the image processing apparatus 700for performing the previewed shift correction on the original,full-size, non-converted images captured by the camera 1100, aspreviously described.

Advantageously, by using the image processing apparatus 500 in thetablet 1600, the tablet 1600 does not have to perform the shiftcorrection on the full-size original images captured by the camera 1600.Rather, the shift correction is performed on smaller-sized, previewimages, thus reducing the amount of processing required. Reducedrequirements for processing are particularly desirable forbattery-powered devices such as the tablet 1600, since it may increasethe life of the battery.

It is noted that in the case where the image processing device 500 iscomprised within the tablet 1600, it may not be necessary for theimage/parameter output 515 to transmit the full-size, original imagesalong with the image effect parameters, φ_(H) or φ_(V), for each image.This is because the full-size, original images may already be stored ina storage device associated with the camera 1100. In this case, it isonly the image effect parameters which must be transmitted, so that theymay be stored with the full-size, original images and processed with thefull-size, original images at a later stage using the imaging processingdevice 700.

It is also noted that the above-described combination of the tablet 1600and image processing device 500 could be used with the interpolatingshift correction embodiment described with reference to FIGS. 19-20. Inthis case, images may be constantly received by the receiver 505 anddisplayed on the display 1630. However, shift correction and parameteroutput are only performed on an image when the image is selected as amarked image using the mark selection box 1910A-C. In this case, theshift correction controller 530 responds to the action of the userselecting the image as marked by generating an image effect parameterφ_(H) or φ_(V) and instructing the image pre-processor 520 to performthe shift correction. In this case, the image effect parameter φ_(H) orφ_(V) is also output by the image/parameter output 515 for storageand/or transmission. For example, the image effect parameter φ_(H) orφ_(V) may be stored and/or transmitted as part of table such as table 2.

It is also envisaged that the components of the image processing device500 could be shared between two devices. Specifically, the receiver 505and converter 510 of the image processing device 500 could be comprisedwithin the camera 1100, and the remaining components could be comprisedwithin the tablet 1600. A captured image could then be converted to asmaller-sized preview image before being transmitted to the tablet 1600.Advantageously, this would allow the preview image to be shift correctedand displayed and the imaging effect parameter φ_(H) or φ_(V) to bestored and/or transmitted whilst, at the same, time reducing thebandwidth required in transmitted image data between the camera 1100 andtablet 1600. It would also further reduce the processing required by thetablet 1600, since the tablet 1600 would no longer have to perform theimage conversion. The battery life of the tablet 1600 would thus befurther improved.

Obviously, numerous modifications and variations of the presentdisclosure are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein.

In so far as embodiments of the invention have been described as beingimplemented, at least in part, by software-controlled data processingapparatus, it will be appreciated that a non-transitory machine-readablemedium carrying such software, such as an optical disk, a magnetic disk,semiconductor memory or the like, is also considered to represent anembodiment of the present disclosure.

Clauses

Features and/or embodiments of the aforesaid disclosure may be generallyexpressed in the following clauses.

1. An image processing device, comprising:

image receiver circuitry configured to receive a first image;

image converter circuitry configured to convert the first image to asecond image, the second image being a smaller-sized representation ofthe first image;

image processor circuitry configured to perform an imaging effect on thesecond image to form a third image, the imaging effect being defined byan imaging effect parameter;

first output device circuitry configured to output the third image fordisplay; and second output device circuitry configured to output thefirst image and the imaging effect parameter.

2. The image processing device according to clause 1, wherein the secondimage is a lower resolution version of the first image.

3. The image processing device according to any preceding clause,wherein the second output device circuitry is configured to output thefirst image and imaging effect parameter to a storage medium.

4. The image processing device according to any preceding clause,wherein the second output device circuitry is configured to transmit thefirst image and the imaging effect parameter to a remote device.

5. The image processing device according to clause 4, wherein the firstimage and imaging effect parameter are transmitted over a computernetwork.

6. The image processing device according to any preceding clause,wherein:

the imaging effect is a shift correction effect and the imaging effectparameter is a shift correction parameter; and

the image processing device comprises shift correction controllercircuitry configured to generate the shift correction parameter.

7. The image processing device according to clause 6, wherein the shiftcorrection parameter is a rotation angle by which the second image isdigitally rotated in three-dimensional space.

8. The image processing device according to clause 7, wherein the shiftcorrection controller circuitry is configured to receive an input from auser to select a value of the angle of the shift correction parameter tobe generated.

9. The image processing device according to clause 7, wherein

the first image is captured by a camera; and

the value of the angle of the shift correction parameter is selected bythe shift correction controller using data from one of an accelerometeror gravimeter associated with the camera.

10. The image processing device according to any preceding clause,wherein:

the image receiver circuitry is configured to receive a plurality offirst images at a predetermined frame rate; and

a time between the image receiver receiving each first image and thefirst output device outputting a respective third image for display isless than or equal to the time period between successive first images.

11. An image processing device comprising

receiver circuitry configured to receive an image and an imaging effectparameter, the imaging effect parameter defining an imaging effect to beperformed on the image;

image processor circuitry configured to perform the imaging effect onthe image, using the imaging effect parameter, to produce a final image;and

final image output device circuitry configured to output the finalimage.

12. The image processing device according to clause 11, wherein thereceiver circuitry is configured to receive the image and imaging effectparameter from a remote device.

13. The image processing device according to clause 12, wherein theimage and imaging effect parameter are received over a computer network.

14. A system comprising:

a first image processing device comprising:

image receiver circuitry configured to receive a first image;

image converter circuitry configured to convert the first image to asecond image, the second image being a smaller-sized representation ofthe first image;

image processor circuitry configured to perform an imaging effect on thesecond image to form a third image, the imaging effect being defined byan imaging effect parameter;

first output device circuitry configured to output the third image fordisplay; and

second output device circuitry configured to output the first image andthe imaging effect parameter; and

a second image processing device comprising:

receiver circuitry configured to receive the first image and the imagingeffect parameter;

image processor circuitry configured to perform the imaging effect onthe first image, using the imaging effect parameter, to produce a finalimage; and

final image output device circuitry configured to output the finalimage.

15. The system according to clause 14, wherein the first and secondimage processing devices are remote from one another, and the secondoutput device of the first image processing device circuitry isconfigured to transmit the first image and imaging effect parameter tothe receiver of the second imaging processing device.16. The system according to clause 15, wherein the first image andimaging effect parameter are transmitted over a computer network.17. An image processing method, comprising:receiving a first image;converting the first image to a second image, the second image being asmaller-sized representation of the first image;performing an imaging effect on the second image to form a third image,the imaging effect being defined by an imaging effect parameter;outputting the third image for display; and outputting the first imageand the imaging effect parameter.18. An image processing method, comprising:receiving an image and an imaging effect parameter, the imaging effectparameter defining an imaging effect to be performed on the image;performing the imaging effect on the image, using the imaging effectparameter, to produce a final image; andoutputting the final image.19. An image processing method comprising:a first step of:

receiving a first image;

converting the first image to a second image, the second image being asmaller-sized representation of the first image;

performing an imaging effect on the second image to form a third image,the imaging effect being defined by an imaging effect parameter;

outputting the third image for display; and

outputting the first image and the imaging effect parameter; and

a second step of:

receiving the first image and the imaging effect parameter;

performing the imaging effect on the first image, using the imagingeffect parameter, to produce a final image; and

outputting the final image.

20. A non-transitory computer readable medium including computer programinstructions, which when executed by a computer causes the computer toperform the method of clause 17.

21. An image processing device, method or system as hereinbeforedescribed with reference to the accompanying drawings.

22. A device for performing shift correction on a sequence of capturedimages, comprising:

image receiver circuitry configured to receive a first image in thesequence, the first image being an image of a scene captured using animage sensor in a first position, and a second image in the sequence,the second image being an image of the scene captured using the imagesensor in a second, different, position; and

image transformer circuitry configured to perform a shift correctiontransformation on each of the first and second images; wherein

the shift correction transformation on the first image is defined by afirst shift correction parameter; and

the shift correction transformation on the second image is defined by asecond shift correction parameter.

23. The device according to clause 22, wherein:

the image receiver circuitry is configured to receive a third image ofthe scene captured using the image sensor as the image sensor is movedfrom the first position to the second position; and

the image transformer circuitry is operable to perform a shiftcorrection transformation on the third image; wherein

the shift correction transformation on the third image is defined by athird shift correction parameter, the third shift correction parameterbeing based on an interpolation of the first and second shift correctionparameters.

24. The device according to clause 23, wherein:

the first and second images of the scene are captured during a firstmovement of the image sensor from the first position to the secondposition; and

the third image is captured during a second movement of the image sensorfrom the first position to the second position, the second movementoccurring subsequent to the first movement.

25. The device according to clause 24, wherein:

the first, second and third images are captured by the image sensor incombination with a lens with a changeable focal length;

the image transformer circuitry is configured to receive the focallength used for the first and second image sensor positions as metadatafrom the lens; and

the image transformer circuitry is configured to correct the first shiftcorrection parameter for a difference between the focal length used atthe first image sensor position during the first image sensor movementand the focal length used at the first image sensor position during thesecond image sensor movement, and to correct the second shift correctionparameter for a difference between the focal length used at the secondimage sensor position during the first image sensor movement and thefocal length used at the second image sensor position during the secondimage sensor movement, the first and second shift correction parametersbeing corrected prior to the generation of the third shift correctionparameter by interpolation.

26. The device according to any of clauses 22-25, wherein the firstshift correction parameter is an angle indicating the amount of theshift correction transformation to be applied to the first image and thesecond shift correction parameter is an angle indicating the amount ofthe shift correction transformation to be applied to the second image.27. The device according to clause 26, comprising user interfacecircuitry configured to receive the angles of the first and second shiftcorrection parameters from a user.28. The device according to any one of clauses 26, comprising:

edge detector circuitry operable to perform edge detection on at least aportion of each of the first and second images to determine one or morestraight lines in each image;

line subset selector circuitry configured to select a subset of one ormore straight lines from the one or more determined straight lines foreach image;

shift correction orientation circuitry configured receive an input fromthe user to determine whether it is a vertical or a horizontal shiftcorrection transformation which is required; and

angle calculator circuitry configured to calculate an angle of each ofthe straight lines in the subset for each image with respect to thehorizontal when it is a horizontal shift correction transformation whichis required or the vertical when it is a vertical shift correctiontransformation which is required.

29. The device according to clause 28, further comprising:

significant line selection circuitry configured to select a mostsignificant line from the subset of one or more straight lines based onone of the length of each line and the proximity of each line to thecentre of the image for each image; wherein

the image transformation circuitry performs the shift correctiontransformation on the first image until the horizontal or vertical angleof the most significant line of the first image is minimised or is madeto be below a first threshold; and

the image transformation circuitry performs the shift correctiontransformation on the second image until the horizontal or verticalangle of the most significant line of the second image is minimised oris made to be below a second threshold.

30. The device according to clause 28, wherein:

the image transformation circuitry performs the shift correctiontransformation on the first image until an average value of thehorizontal or vertical angles of the straight lines in the subset of thefirst image is minimised or is made to be below a first threshold; and

the image transformation circuitry performs the shift correctiontransformation on the second image until an average value of thehorizontal or vertical angles of the straight lines in the subset of thesecond image is minimised or is made to be below a second threshold.

31. The device according to any one of clauses 28-30, wherein:

the line subset selector circuitry is configured to be responsive to aninput from a user in order to select the subset of straight lines ineach of the first and second images.

32. The device according to clause 28-30, wherein:

the line subset selector circuitry is configured to:

determine a first set of straight lines for each image, wherein eachstraight line in the first set exceeds a predetermined threshold length;

determine a second set of straight lines from the first set for eachimage, wherein each straight line in the second set has a horizontal orvertical angle below a predetermined threshold angle; and

determine the subset of straight lines for each image from the secondset based on one of the length of each line and the proximity of eachline to the centre of the image.

33. The device according to any one of clause 22 to 32 furthercomprising the image sensor which, in use, captures the first and secondimages.

34. The device according to any one of clause 22 to 33, wherein theshift correction transformation is performed on a low resolution versionof the first and second images, and the device further comprises outputcircuitry operable to transfer the first and second shift correctionparameter over a network.35. A system comprising the device according to clause 34 connected, inuse, over a network to a server, wherein the server comprises a receivercircuitry configured to receive the first and second shift correctionparameter and the full resolution version of the first and second imageand an image processor circuitry configured to perform shifttransformation correction on the received full resolution version of thefirst and second image using the received first and second shiftcorrection parameter.36. A method of performing shift correction on a sequence of capturedimages, comprising:

receiving a first image in the sequence, the first image being an imageof a scene captured using an image sensor in a first position, and asecond image in the sequence, the second image being an image of thescene captured using the image sensor in a second, different, position;and

performing a shift correction transformation on each of the first andsecond images; wherein

the shift correction transformation on the first image is defined by afirst shift correction parameter; and

the shift correction transformation on the second image is defined by asecond shift correction parameter.

37. A non-transitory computer readable medium including computer programinstructions, which when executed by a computer causes the computer toperform the method of clause 36.

The invention claimed is:
 1. An image processing device, comprising:image receiver circuitry configured to receive a first image; imageconverter circuitry configured to convert the first image to a secondimage, the second image being a smaller-sized representation of thefirst image; image processor circuitry configured to perform an imagingeffect on the second image to form a third image, the imaging effectbeing defined by an imaging effect parameter; first output devicecircuitry configured to output the third image for display before thefirst image is captured for storage; and second output device circuitryconfigured to output the first image and the imaging effect parameterafter the third image is output for display, wherein a time between theimage receiver receiving the first image and the first output deviceoutputting a respective third image for display is less than or equal toa frame rate at which the image receiver receives a plurality of firstimages.
 2. The image processing device according to claim 1, wherein thesecond image is a lower resolution version of the first image.
 3. Theimage processing device according to claim 1, wherein the second outputdevice circuitry is configured to output the first image and imagingeffect parameter to a storage medium.
 4. The image processing deviceaccording to claim 1, wherein the second output device circuitry isconfigured to transmit the first image and the imaging effect parameterto a remote device.
 5. The image processing device according to claim 4,wherein the first image and imaging effect parameter are transmittedover a computer network.
 6. The image processing device according toclaim 1, wherein: the imaging effect is a shift correction effect andthe imaging effect parameter is a shift correction parameter; and theimage processing device comprises shift correction controller circuitryconfigured to generate the shift correction parameter.
 7. The imageprocessing device according to claim 6, wherein the shift correctionparameter is a rotation angle by which the second image is digitallyrotated in three-dimensional space.
 8. The image processing deviceaccording to claim 7, wherein the shift correction controller circuitryis configured to receive an input from a user to select a value of theangle of the shift correction parameter to be generated.
 9. The imageprocessing device according to claim 7, wherein the first image iscaptured by a camera; and the value of the angle of the shift correctionparameter is selected by the shift correction controller using data fromone of an accelerometer or gravimeter associated with the camera.
 10. Animage processing method, comprising: receiving a first image; convertingthe first image to a second image, the second image being asmaller-sized representation of the first image; performing an imagingeffect on the second image to form a third image, the imaging effectbeing defined by an imaging effect parameter; outputting the third imagefor display; and outputting the first image and the imaging effectparameter, wherein a plurality of first images are received at a firstframe rate, and a time between receiving each first image and outputtinga respective third image for display is less than or equal to the timeperiod between successive first images.
 11. A non-transitory computerreadable medium including computer program instructions, which whenexecuted by a computer causes the computer to perform the method ofclaim
 10. 12. An image processing device comprising: circuitryconfigured to receive a first image; convert the first image to a secondimage, the second image being a smaller-sized representation of thefirst image; perform an imaging effect on the second image to form athird image, the imaging effect being defined by an imaging effectparameter; output the third image for display; and output the firstimage and the imaging effect parameter, wherein a plurality of firstimages are received at a first frame rate, and a time between receivingeach first image and outputting a respective third image for display isless than or equal to the time period between successive first images.13. The image processing device according to claim 12, wherein thesecond image is a lower resolution version of the first image.
 14. Theimage processing device according to claim 12, wherein the first imageand the imaging effect parameter are output to a storage medium.
 15. Theimage processing device according to claim 12, wherein the first imageand the imaging effect parameter are output to a remote device.
 16. Theimage processing device according to claim 12, wherein: the imagingeffect is a shift correction effect and the imaging effect parameter isa shift correction parameter; and the circuitry is configured togenerate the shift correction parameter.
 17. The image processing deviceaccording to claim 16, wherein the shift correction parameter is arotation angle by which the second image is digitally rotated inthree-dimensional space.
 18. The image processing device according toclaim 17, wherein the circuitry is configured to receive an input from auser to select a value of the angle of the shift correction parameter tobe generated.